Interaction method, electronic device, and computer storage medium

Abstract

The present application relates to the technical field of terminal devices, and provides an interaction method, an electronic device, and a computer storage medium. The method enables display of content corresponding to user actions in an always-on display state, increasing interactive enjoyment in always-on display scenarios. The method can be applied to an electronic device, and specifically comprises: determining that a screen of the electronic device is in an always-on display state; detecting a first action of a user; and, in response to the first action, displaying first content in the always-on display state, wherein the first content corresponds to the first action.

Description

Interaction methods, electronic devices and computer storage media

[0001] This application claims priority to Chinese Patent Application No. 202410705862.8, filed on May 31, 2024, entitled "Interactive Method, Electronic Device and Computer Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of terminal equipment technology, and in particular to an interaction method, electronic device, and computer storage medium. Background Technology

[0003] Electronic devices that support Always On Display (AOD) can still display content such as time, date, weather, or wallpaper even when the screen is off.

[0004] AOD technology meets the user's need to quickly obtain information such as time in screen-off scenarios, but it does not support interaction with the user and cannot meet the user's possible interaction needs in screen-off scenarios. Summary of the Invention

[0005] This application discloses an interaction method, an electronic device, and a computer storage medium, which can display corresponding content based on the user's actions when the electronic device is in a screen-off state, interact with the user, and improve the user experience.

[0006] In a first aspect, this application provides an interaction method applied to an electronic device, the method comprising: determining that the screen of the electronic device is in a screen-off state; detecting a first action of a user; and, in response to the first action, displaying first content in the screen-off state; wherein the first content corresponds to the first action.

[0007] The screen-off state refers to the state of an electronic device from when the screen is turned off until the user wakes the screen. This state can include screen-off, partial display, or full-screen dimming. Screen-off means the screen is completely off, and all pixels on the display do not emit light; this includes both active and passive light emission. Partial display means some pixels on the screen emit light, while the rest do not. Full-screen dimming means all pixels on the screen emit light, but after dimming or brightness reduction, the brightness of all pixels is below a preset threshold.

[0008] In screen-off scenarios, the above method displays content corresponding to the user's action after the user performs an action, supports interaction with the user, meets the user's interaction needs in screen-off scenarios, and enriches the user experience.

[0009] In one possible implementation, displaying the first content includes: displaying the first handwriting; the first handwriting corresponds to the first action.

[0010] The above method displays handwriting corresponding to the user's action when the screen is off. For example, if the user's finger makes a writing or drawing motion on the screen, the screen displays handwriting corresponding to the action, allowing the user to experience the interactive fun of "writing" on the screen when the electronic device is off, increasing the fun.

[0011] In one possible implementation, before displaying the first handwriting, the method further includes: displaying a background in a screen-off state; the background including wallpaper and / or specified information; displaying the first handwriting in a screen-off state includes: displaying the background and the first handwriting in a screen-off state.

[0012] The background can include wallpaper, which can be static or dynamic, or a full-screen wallpaper or a partial wallpaper. Specified information can include one or more of the following: time, date (including Gregorian and / or Lunar calendar dates), weather, battery level, notification messages, etc. The background and the first handwriting are displayed. The first handwriting and the background can be displayed independently on the screen, or the first handwriting can be displayed over the background. For example, the background can be a full-screen wallpaper, and the handwriting can be displayed over the full-screen wallpaper; alternatively, the background can be a partial wallpaper (referred to as layout wallpaper) and information such as time, and the handwriting can be displayed separately from the partial wallpaper, or partially or completely over the partial wallpaper.

[0013] The above method can display handwriting in the background of always-on display and is compatible with existing AOD technology.

[0014] In one possible implementation, the background includes a first wallpaper, and the first handwriting is displayed on the first wallpaper; wherein the first handwriting is transparent; and the area on the first wallpaper covered by the first handwriting is displayed through the first handwriting.

[0015] The first stroke is transparent, which can be understood as the first stroke being displayed with a predetermined transparency, such as the first stroke being displayed with a second transparency.

[0016] In the above method, in a scenario where the screen is fully darkened and wallpaper is displayed, the user can perform actions such as writing, drawing, or swiping on the screen. The resulting handwriting is transparent, and the area of ​​the wallpaper at the bottom covered by the handwriting can be seen through the handwriting, presenting a hollow display effect.

[0017] In one possible implementation, the background includes a first wallpaper; displaying the background can be done by displaying the first wallpaper with a first transparency; displaying the background and the first handwriting can be done by displaying the first handwriting on the first wallpaper. The first handwriting is displayed with a second transparency, which is different from the first transparency.

[0018] In one possible implementation, the second transparency is less than the first transparency.

[0019] The first wallpaper, displayed with a first transparency, is obtained by overlaying a first overlay on the first wallpaper; each pixel in the first overlay has a first transparency. The first handwriting, displayed with a second transparency, is obtained by overlaying a second overlay on the first wallpaper; the second overlay is obtained by mixing the first overlay with the first handwriting; the pixels corresponding to the first handwriting in the second overlay have a second transparency.

[0020] In the above method, the second transparency is less than the first transparency. This can be achieved in a scenario where the screen is darkened and wallpaper is displayed. When the user makes an action, the displayed handwriting becomes transparent and its transparency (second transparency) is less than the transparency (first transparency) of the wallpaper that was originally darkened. This allows the handwriting to show through the darkened wallpaper in the area covered by the handwriting, creating an effect where the area where the handwriting passed is brightened, giving the user an interactive experience similar to "wiping glass".

[0021] In one possible implementation, the background includes a first wallpaper; displaying the background can be done by displaying the first wallpaper at a first brightness. Displaying the background and the first handwriting can be done by displaying the first handwriting on the first wallpaper; wherein the first handwriting is displayed at a second brightness; the second brightness is different from the first brightness.

[0022] In the above method, the first wallpaper can be a full-screen wallpaper, the first handwriting is displayed on top of the first wallpaper, and the first handwriting and the first wallpaper are displayed with different brightness levels, so that the handwriting can be displayed against the background of the wallpaper. The adjustment of the first brightness and the second brightness can be achieved by overlaying a transparent overlay and controlling the transparency of pixels at corresponding positions on the overlay to achieve different brightness levels; or by directly controlling the brightness of pixels at corresponding positions, for example, setting the brightness value of the pixels in the area corresponding to the handwriting to be different from the brightness value of the pixels in the area of ​​the wallpaper not covered by the handwriting.

[0023] In one possible implementation, displaying the first handwriting includes: displaying the first handwriting with a specified style.

[0024] For example, the handwriting style can be specified by the user, such as determining a first style in response to a first user action; wherein the first action includes selecting the first style from at least one style candidate. The at least one style candidate may include styles such as ink wash, stylus, sand painting, crayon, etc.

[0025] In one possible implementation, displaying a first handwriting with a specified style includes: displaying a first handwriting comprising multiple stroke elements; the multiple stroke elements being brush elements conforming to the specified style.

[0026] In one possible implementation, displaying the first handwriting includes: displaying a first handwriting with varying line thickness; wherein the line thickness is determined based on the speed and / or pressure corresponding to the first action.

[0027] One possible implementation involves displaying the first handwriting, including displaying a first handwriting with a constant line thickness.

[0028] In one possible implementation, displaying the first handwriting includes: displaying the first handwriting including the first stroke; wherein the starting point and / or ending point of the first stroke is displayed in a first manner; the middle part of the first stroke is displayed in a second manner, the first manner being different from the second manner; wherein the middle part is the stroke between the starting point and the ending point of the first stroke.

[0029] In one possible implementation, a first handwriting including a first stroke is displayed; wherein the first stroke includes multiple first stroke elements, the distribution direction of the multiple first stroke elements is determined according to the direction in which the user makes the first stroke action on the screen; the first action includes a first stroke action.

[0030] In one possible implementation, the style is specified as one of a stylus, sand art, or crayon; displaying a first handwriting with the specified style includes: displaying a first handwriting with the specified style; the first handwriting includes a first stroke of uniform thickness.

[0031] In one possible implementation, the style is specified as sand art or crayon; displaying a first handwriting comprising multiple stroke elements includes: displaying a first stroke element and / or a second stroke element; wherein the multiple stroke elements include a first stroke element and / or a second stroke element; the first handwriting comprises multiple sampling points; the first stroke element is obtained by rotating an original stroke element; the second stroke element is obtained by jittering the position of the original stroke element relative to a sampling point.

[0032] In a second aspect, this application provides an electronic device, including a memory for storing a computer program; and a processor for executing the computer program in the memory to perform the interactive method provided by the first aspect of this application and any implementation thereof.

[0033] Thirdly, this application provides a computer storage medium storing a computer program that, when executed by a processor, implements the interaction method provided by the first aspect of this application and any implementation thereof.

[0034] Fourthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the interaction method provided by the first aspect of this application and any implementation thereof.

[0035] Fifthly, this application provides an electronic device that includes the methods or apparatus described in any aspect or embodiment of this application. The aforementioned electronic device is, for example, a chip.

[0036] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this application do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this application can be combined in any suitable manner. Those skilled in the art will understand that this application can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0037] The following describes the accompanying drawings used in this application.

[0038] Figure 1 is a schematic diagram of the hardware structure of an electronic device provided in this application;

[0039] Figure 2 is a schematic diagram of the hardware architecture of a sensor hub deployed in an electronic device according to this application;

[0040] Figure 3 is a schematic diagram of the software architecture of a sensor hub deployed in an electronic device according to this application;

[0041] Figure 4 is a schematic diagram of the software layered architecture on the AP side of an electronic device provided in this application;

[0042] Figure 5 is an example diagram of an interaction method provided in this application in a practical application scenario;

[0043] Figures 6a to 6c are example diagrams of the three states included in the screen-off state in the interaction method provided in this application;

[0044] Figures 7a to 7h are multiple example diagrams showing the handwriting displayed by the interactive method provided in this application in a real-world application scenario;

[0045] Figure 8 is an example of a user interface for setting screen-off interaction in an interaction method provided in this application;

[0046] Figure 9 is a flowchart illustrating an embodiment of an interaction method provided in this application;

[0047] Figure 10 is a schematic diagram of generating handwriting based on action data in one embodiment of an interaction method provided in this application;

[0048] Figure 11 is a schematic diagram of brush elements with different styles in one embodiment of an interaction method provided in this application;

[0049] Figure 12 is a schematic diagram of generating a corresponding Bézier curve based on the user's action trajectory in one embodiment of an interaction method provided in this application;

[0050] Figure 13 is a schematic diagram of the process framework for generating handwriting using a stylus, sand painting, and crayon styles in one embodiment of an interactive method provided in this application.

[0051] Figure 14 is a schematic diagram of the process framework for random rotation and position jitter processing when sand painting and crayon generate handwriting in an embodiment of an interactive method provided in this application.

[0052] Figure 15 is a schematic diagram of random rotation during handwriting generation in one embodiment of an interactive method provided in this application;

[0053] Figure 16 is a schematic diagram of position jitter when generating handwriting in one embodiment of an interactive method provided in this application;

[0054] Figure 17 is a schematic diagram of the process framework for generating handwriting in the ink wash style in one embodiment of an interactive method provided in this application;

[0055] Figure 18 is a schematic diagram showing how the direction of the brush element changes with the direction of the pen stroke in one embodiment of an interaction method provided in this application;

[0056] Figure 19 is a flowchart illustrating layer blending processing in one embodiment of an interactive method provided in this application.

[0057] Figure 20 is an example interface diagram showing the display effect of handwriting gradually disappearing over time in one embodiment of an interactive method provided in this application;

[0058] Figure 21 is a flowchart illustrating an interaction method provided in this application. Detailed Implementation

[0059] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of, and not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0060] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more.

[0061] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0062] In the embodiments of this application, the words "exemplary" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options.

[0063] Many electronic devices support Always-on Display (AOD), which means that even when the screen is off, information such as time, date, battery level, weather, or notification messages are still displayed, allowing users to know the time and other information without waking up the electronic device; some models can also support displaying static wallpapers or animated wallpapers, enriching the user's visual experience.

[0064] Currently, most electronic devices only support screen-off display in screen-off scenarios and do not support interaction. This means that users can only see the information and / or wallpaper displayed in the screen-off state, without providing interactive feedback, and cannot meet users' interactive needs in screen-off scenarios.

[0065] This application proposes an interaction method that can be applied to electronic devices. When the electronic device is in a screen-off state, it can detect whether the user has made an action and display content corresponding to the action on the screen. For example, the user's action may be writing or drawing on the screen with their finger, and the displayed content may be handwriting, which may be a graphic or stroke, etc., so that the user can experience the interactive fun of writing or drawing on the screen when the electronic device is in a screen-off state.

[0066] In some embodiments, displaying content corresponding to the action can mimic the user's action trajectory and display content that is as consistent as possible with the user's action trajectory. For example, following the trajectory of the user's swiping action on the screen, displaying graphics or text that are consistent with the user's action trajectory. The display method of text or graphics can simulate a cutout effect, providing an interactive experience similar to writing or drawing on a flat surface such as foggy glass, satisfying the user's interactive needs in screen-off scenarios, and increasing fun.

[0067] In other embodiments, displaying content corresponding to an action can mean displaying content that is mapped to the user's action, and the displayed content may not be consistent with the user's action. For example, a mapping relationship between actions and content can be pre-established. When the screen is off, if the user draws different shapes or makes different touch actions on the screen, the content corresponding to each shape or touch action will be displayed. For example, if the user draws a triangle on the screen, a preset dynamic or static wallpaper that is mapped to the triangle action will be displayed. Another example is when the user performs a specified touch action such as swiping down, swiping left, swiping right, swiping up, or triple-tapping while the screen is off; the screen can automatically display a dynamic wallpaper, such as a fireworks animation wallpaper or a deer animation wallpaper.

[0068] The interaction method provided in this application can solve the technical problem of not supporting interaction with users in the screen-off state, meet the user's interaction needs in the screen-off scenario, add interactive fun, and improve user experience.

[0069] It should be noted that in some embodiments, the user can perform actions without making them on the screen, but rather within a specified distance around the electronic device (e.g., within the machine vision range of the electronic device), making the actions while suspended in the air. The electronic device recognizes the user's actions (which may include gestures) and displays corresponding content on the screen. For example, image data collected by devices such as image acquisition devices of the electronic device can be processed using appropriate software algorithms to obtain the recognition result of the user's actions, and then display the content corresponding to the recognized actions. For example, the image acquisition device could be a front-facing camera, an infrared sensor, etc.

[0070] It should be noted that in some embodiments, after displaying content corresponding to the user's action, such as after displaying handwriting, if no new action is detected from the user within a predetermined period of time, the currently displayed handwriting can be gradually removed from the screen. For example, a few seconds after the user's finger is removed, the strokes or graphics gradually disappear, restoring the original state before the handwriting was displayed.

[0071] In this application, the electronic device can be a mobile phone, tablet computer, handheld computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), as well as smart home devices such as smart TVs and smart refrigerators with displays, wearable devices such as smart bracelets and smartwatches with displays, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) with displays, in-vehicle devices or smart city devices with displays. The embodiments of this application do not impose special restrictions on the specific type of electronic device.

[0072] Figure 1 illustrates a schematic diagram of the hardware structure of an electronic device 100.

[0073] It should be understood that the electronic device 100 shown in Figure 1 is merely an example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in Figure 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0074] As shown in Figure 1, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0075] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0076] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In another embodiment, touch sensor 180K can also be located on the surface of electronic device 100, in a different position than display screen 194.

[0077] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0078] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0079] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0080] In one embodiment, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0081] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141. The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc.

[0082] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In another embodiment, the antenna can be used in conjunction with a tuning switch.

[0083] The mobile communication module 150 can provide wireless communication solutions for applications on the electronic device 100, including second-generation (2G), third-generation (3G), fourth-generation (4G), fifth-generation (5G), and sixth-generation (6G) mobile communication technologies. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0084] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment, the modem processor may be a separate device. In another embodiment, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0085] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0086] In one embodiment, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0087] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0088] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0089] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0090] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, etc. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment, the ISP can be set in the camera 193.

[0091] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0092] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0093] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0094] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0095] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment, the audio module 170 can be located in the processor 110, or some functional modules of the audio module 170 can be located in the processor 110.

[0096] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0097] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0098] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may be equipped with at least one microphone 170C.

[0099] The 170D headphone jack is used to connect wired headphones.

[0100] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0101] Motor 191 can generate vibration alerts.

[0102] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0103] The SIM card interface 195 is used to connect the SIM card.

[0104] In some embodiments, the interaction method proposed in this application can be executed independently by the smart sensor hub without waking up the AP. That is, the interaction method provided in this application can achieve interaction with the user through the sensor hub while the AP is in sleep mode. Specifically, the sensor hub is used to connect to and process the data collected by the sensors in the sensor module 180.

[0105] A sensor hub is a sensor management component that enables hardware abstraction, management, and data processing of sensor devices such as touch sensors. A sensor hub combines a low-power microcontroller unit (MCU) with a lightweight real-time operating system (RTOS) to reduce the power consumption of electronic devices when connecting to and processing sensor data.

[0106] The sensor hub that may be used in the embodiments of this application will be described below from the perspectives of hardware implementation and software implementation.

[0107] As shown in Figure 2, in the hardware architecture using Sensor Hub, the electronic device also includes an MCU, and the processor 110 may include an application processor (AP). The AP runs a general-purpose operating system, such as Android or HarmonyOS. The MCU runs a lightweight RTOS. The MCU is the hardware foundation for implementing Sensor Hub; in other words, the hardware chip of Sensor Hub can be an MCU.

[0108] In this application, the sensor hub supports multiple hardware architectures, such as one of the following three architectures: MCU-integrated, MCU-external, and MCU-independent. Figure 2 shows an example of the MCU-independent hardware architecture. In the MCU-independent architecture, the MCU is set up as a separate chip, which can be understood as the sensor hub being a separate chip placed between the AP and the sensors. In the MCU-integrated architecture, the MCU can be integrated into the AP, which can be understood as the AP integrating the sensor hub. Various sensors first provide data to the sensor hub inside the AP, and the sensor hub then fuses the data and provides it to the AP. In the MCU-external architecture, the MCU can be designed as a peripheral device, integrated with the sensors, which can be understood as combining the sensor hub and sensors into one.

[0109] The sensors shown in Figure 2 may include one or more of the sensors shown in sensor module 180 in Figure 1, or may include other sensors. One or more of the sensors may be micro-electro-mechanical system (MEMS) sensors.

[0110] In terms of software, there are several options for the software architecture implementation of a sensor hub. For example, a sensor hub can adopt a layered design, including a sensor hub logic implementation layer and a sensor driver layer. The sensor hub logic implementation layer provides a unified sensing interaction interface to the application (AP side), a unified adaptation interface for sensors, and various common acquisition strategies to the application (AP side). The sensor driver layer includes drivers for each sensor and a registration interface for the AP-side application to mount onto the sensor hub.

[0111] In some embodiments of this application, an interaction module can be deployed in the sensor logic implementation layer. This interaction module can simulate an application and read the data collected by the sensor by calling the sensor interaction interface. For example, after the screen enters the screen-off state, the sensor interaction interface of the sensor logic implementation layer can be used to read the data collected by the touch sensor 180K, identify the user's actions based on the collected data, and configure the current collection strategy based on the various collection strategies provided by the sensor logic implementation layer, such as setting the time interval of sampling points (sampling period).

[0112] Alternatively, in some embodiments, the software architecture of the Sensor hub can adopt the LiteOS sensing framework. As shown in Figure 3, the LiteOS sensing framework includes a Sensor Manager layer, a BSP manager layer, and a Converged Algorithms layer. The BSP (Board Support Package) refers to the board-level upgrade package. The Sensor Manager layer is used to implement unified sensor interaction management, such as sensor configuration, sampling, and reporting. The BSP Manager layer provides a unified driver interface and is responsible for sensor driver management and sensor interaction management, such as sensor opening, closing, reading, writing, and data updating. The Converged Algorithms layer deploys a fusion algorithm library for algorithm fusion on the MCU according to specific business requirements.

[0113] As shown in Figure 3, when the Sensor Hub uses the LiteOS sensing framework, an interaction module can be deployed in the Converged Algorithms layer. This interaction module performs operations such as enabling, reading, and writing to the sensor through the driver interface provided by the BSP Manager layer. For example, after the electronic device's screen is off, the touch sensor may be in a turned-off state, meaning it does not detect any touch activity. In the interaction method provided in this application, after entering the screen-off state, the touch sensor 180K can be enabled based on the driver interface provided by the BSP layer to perform touch detection in the screen-off state. The sampling period of the touch sensor 180K can be configured using the interaction management interface provided by the Sensor Manager layer.

[0114] Interactive modules deployed in the sensor hub logic implementation layer or Converged Algorithms layer can generate corresponding handwriting based on the data collected by the sensors, and then display the handwriting on the display screen 194.

[0115] It should be noted that, in some embodiments, due to the high power consumption of the AP during operation, the interaction method provided in this application can be executed by the sensor hub in the AP sleep state, so as to reduce the power consumption required to implement the interaction method provided in this application in the screen-off state.

[0116] In another embodiment, the interaction method provided in this application can also be executed by the AP by waking it up.

[0117] In the embodiment where the interaction is performed by the sensor hub while the AP is in sleep mode, the GPU does not need to enter sleep mode with the AP. Therefore, while the AP is in sleep mode, the interaction module in the sensor hub can communicate with the GPU to transmit the generated image data containing handwriting (e.g., the image data after the full-screen overlay 03 and wallpaper 04 are mixed in Figure 19) to the GPU. The GPU performs mathematical and geometric calculations and performs graphics rendering. The GPU is connected to the display screen 194 and can transmit the rendered image data to the display screen 194 for display. Implementing the interaction method proposed in this application while the AP is in sleep mode can modify the software architecture of the sensor hub, for example, by adding an interaction module to implement handwriting display through the GPU.

[0118] In the embodiment where the interaction method proposed in this application is executed by the AP, the electronic device can realize the handwriting display function through the GPU, display screen 194, and application processor AP. The GPU is connected to the display screen 194 and the application processor AP. It should be noted that since the AP does not sleep in this embodiment, an interaction module does not need to be set in the Sensor Hub, that is, an interaction module does not need to be added to the Sensor Hub software architecture, and the interaction method can be implemented based on the corresponding software architecture on the AP side.

[0119] The application processor (AP) in the electronic device 100 runs a general-purpose operating system. The software architecture of the general-purpose operating system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example for illustration.

[0120] Figure 4 illustrates an exemplary schematic diagram of the software architecture corresponding to the AP in the electronic device 100.

[0121] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In one implementation, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0122] The application layer can include a series of application packages.

[0123] As shown in Figure 4, the application package may include applications such as camera, calendar, music, gallery, SMS, call, navigation, settings, and browser. The interaction module in this embodiment can be a standalone application or a functional module integrated into other applications; this application does not limit this. The application in this application can also be replaced with other forms of software such as mini-programs or atomic services.

[0124] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications within the application layer. The application framework layer includes predefined functions. For example, in the interaction method proposed in this application, the operation of generating handwriting based on user action data can be implemented through an algorithm module, which can be added to the application framework layer as an API.

[0125] Specifically, as shown in Figure 4, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager. The window manager manages window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to the application. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, and a phone book. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application. The display interface can consist of one or more views. For example, a display interface including a text notification icon may include a view for displaying text and a view for displaying images. The phone manager provides communication functions for the electronic device 100, such as managing call status (including connection, hang-up, etc.). The resource manager provides the application with various resources, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows the application to display notification information in the status bar, which can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify users of download completion, message alerts, etc. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, vibrating electronic devices, and flashing indicator lights.

[0126] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0127] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0128] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0129] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0130] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0131] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0132] A 2D graphics engine is a graphics engine for 2D drawing.

[0133] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0134] It should be understood that the above only uses the Android system as an example. The method of this application can also be applied to non-Android systems, such as iOS operating system, HarmonyOS operating system, ColorOS operating system and other types of operating systems.

[0135] The following example illustrates the workflow of software and hardware that execute interaction methods based on the AP, using an interaction scenario in screen-off mode as an example.

[0136] After the screen of an electronic device is turned off, it enters an always-on display (AOD) state, such as a partial display based on AOD or a full-screen dimmed state. When the touch sensor 180K receives a user's touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into raw input events (including touch coordinates, touch operation timestamps, etc.). The raw input events are stored in the kernel layer. The application framework layer obtains the raw input events from the kernel layer and determines the events to be triggered based on them. For example, the event to be triggered is to generate and display content corresponding to the user's action (e.g., handwriting). The application framework layer generates corresponding image data or other format data based on the data collected by the touch sensor 180K, and controls the display driver by calling the kernel layer to display the content corresponding to the user's action on the display screen 194.

[0137] The following describes the application scenarios involved in the embodiments of this application and examples of user interfaces in these scenarios. The examples below primarily use handwriting as an example for illustration.

[0138] Figure 5 illustrates a schematic diagram of a user interface for an interactive scenario.

[0139] Taking a mobile phone as an example, as shown in Figure 5, even after the phone screen is off, the AOD (Always On Display) technology still displays the time information "10:49" and the date information "July 12th, Saturday, the third day of the eighth lunar month". In this state, the phone's operating system (general operating system or real-time operating system) detects that the screen is off and activates the touchscreen sensor 180K for touch detection, as shown in interface 10a. When the user begins to perform actions on the screen, the touch sensor 180K generates a handwriting corresponding to the user's actions and drives the display screen to display the handwriting. The generated handwriting can be part or all of a graphic, text, or symbol, as shown in interface 10b in Figure 5. Following the user's swiping motion on the screen, the display screen displays the corresponding graphic in real time. In interface 10c, when the user's swiping motion ends, the display screen displays the complete graphic.

[0140] The heart-shaped graphic shown in Figure 5 is only an example. The interactive method provided in this application embodiment supports the display of various regular or irregular shapes, graphics, text, symbols, etc.

[0141] It should be noted that in the various embodiments of this application, the screen-off state or screen-off scenario can specifically be one or more combinations of states such as screen off, partial display, or full-screen dimming. As shown in Figure 6a, the screen-off state, also known as the screen-off state, can be a state in which all pixels of the screen do not emit light, or a state in which the screen's backlight is off. For screens that support independent light emission of pixels, such as Organic Light Emitting Diode (OLED) screens, the screen-off state can be a state in which all pixels of the screen do not emit light (are turned off), that is, a visually completely black screen state; for Liquid Crystal Display (LCD) screens, the screen-off state refers to a state in which the backlight (backlight lamp) does not emit light (is off).

[0142] Partial display mode refers to a state where some pixels on the screen are lit while others are off. For example, after the screen enters the Always-On Display (AOD) state, based on AOD technology, pixels used to display information such as time, date, and weather are lit, or some pixels used to display dynamic or static wallpapers are lit, while pixels in other locations are not illuminated (also referred to as off or turned off). For example, as shown in Figure 6b, the pixels used to display the time "10:49" and the pixels used to display the date "July 12th, Saturday, August 3rd" are lit, while the remaining pixels are off (also referred to as not illuminating or turned off). This state is an example of partial display.

[0143] Full-screen dimming refers to a state where all pixels on the screen are illuminated, but their brightness is dimmed or reduced below a preset threshold. For example, full-screen dimming could be a display state where the wallpaper or interface is dimmed, or a display state where the backlight brightness is reduced. For instance, on an OLED screen, after the screen is off, based on AOD (Always-On Display) technology, to reduce power consumption when displaying information such as the time and / or wallpaper, all pixels on the screen are dimmed before displaying, resulting in a full-screen dimming effect. For example, as shown in interface 11c in Figure 6c, after the screen is off, the time, date, and wallpaper 111 are still displayed, and all pixels on the screen are illuminated. To reduce power consumption, the screen is dimmed; this is an example of a full-screen dimming state. For LCD screens, it could be a state where the backlight brightness is reduced. For example, before electronic devices such as mobile phones enter a screen-off state, the backlight brightness can be automatically reduced; this state can also be considered a full-screen dimming state.

[0144] It should be noted that partial display mode can include partial dimming, where the areas of the screen where the luminous pixels are displayed are dimmed to further reduce power consumption. For example, partial display mode could be a scenario where a wallpaper is partially displayed, and the displayed wallpaper is dimmed, resulting in a partially dimmed state.

[0145] In addition, the full-screen dimming state can include either partially displaying the wallpaper or displaying the wallpaper in full screen.

[0146] Darkening in full-screen mode can be achieved using the Multiply blending mode. For example, a semi-transparent black overlay can be applied to the wallpaper under Multiply mode. This overlay can be a layer containing pixel values ​​with four dimensions: RGB and opacity. All pixels maintain a consistent RGB channel value and a uniform opacity α, where 0 ≤ α ≤ 100. When α = 0, the pixel opacity is highest (completely transparent), displaying the wallpaper's original color. When α = 100, the opacity is lowest (completely opaque), obscuring the corresponding pixels on the wallpaper. Adjusting the value of α adjusts the brightness and darkness of the wallpaper.

[0147] The example shown in Figure 5 can be a full-screen darkened state after darkening processing, that is, the screen displays a black semi-transparent overlay. When the user makes a heart-shaped gesture on the screen, the phone screen displays the heart-shaped strokes in real time and shows them on the screen. Specifically, the area corresponding to the strokes in the black semi-transparent overlay is erased to present a hollow effect, simulating the interactive effect of wiping glass.

[0148] The interactive method provided in this application supports displaying handwriting in various styles. For example, in some embodiments, multiple styles are provided, such as ink painting style, sand painting style, crayon style, and stylus style.

[0149] In other embodiments, different art styles, calligraphy styles, or fonts can be provided. For example, art styles may include abstract, realist, impressionist, and ink painting styles, automatically displaying images or graphics of the corresponding style based on the user's drawing. Calligraphy styles may include different fonts, automatically displaying text in the corresponding font or style based on the user's handwritten text on the screen. For example, automatically displaying regular script, Song typeface, and seal script fonts.

[0150] The following examples illustrate the techniques used: ink painting style, sand painting style, crayon style, and stylus style.

[0151] Referring to FIG. 7a, FIG. 7a shows an example interface displaying ink-wash style handwriting. In the example illustratively shown in FIG. 7a, after the screen is turned off, based on the AOD technology, it enters the partial display state, where some pixel points on the screen emit light and the rest do not. The glowing area is used to display the wallpaper 12a2, and there is a layer of black semi-transparent mask on the surface of the wallpaper 12a2, presenting a dimmed wallpaper effect. The user makes an action of writing the Chinese character "Li" (pinyin: li) on the screen. After the electronic device determines that the current style is the ink-wash style, according to the user's action, in accordance with the handwriting generation method corresponding to the ink-wash style, a corresponding handwriting layer is generated. The handwriting layer carries the coordinate information of the multiple pixel points forming the handwriting in the screen. The handwriting layer and the black semi-transparent mask are subjected to subtractive mixing to obtain the black semi-transparent mask after erasing the handwriting, and the display screen is driven to display, presenting the ink-wash style handwriting 12a1, that is, the Chinese character "Li". In this way, after the screen is turned off, the user can write on the screen, and the written words can be displayed in a hollowed-out manner, revealing the underlying wallpaper, enriching the visual experience, adding interactive fun, and can also play a role in transmitting information in some scenarios. For example, in a relatively quiet venue, library, etc., information can be transmitted by writing on the screen.

[0152] Referring to FIGS. 7b to 7d, FIG. 7b shows an example interface displaying handwriting pen style handwriting. FIG. 7c shows an example interface of sand painting style handwriting, and FIG. 7d shows an example interface of crayon style handwriting.

[0153] In the example illustratively shown in FIG. 7b, after the screen is turned off, based on AOD, it enters the partial display state, and some pixel points are lit to display the wallpaper 121. The wallpaper 121 has been subjected to a dimming treatment and is covered with a layer of black semi-transparent mask on the surface. The user makes an action of drawing a heart on the screen. The electronic device determines that the current style is the handwriting pen style, and according to the user's action, in accordance with the handwriting generation method corresponding to the handwriting pen style, a corresponding handwriting layer is generated. The handwriting layer and the black semi-transparent mask are subjected to subtractive mixing, and the display screen is driven to display, presenting a heart-shaped graphic 12b with the handwriting pen style, simulating the effect of wiping the glass fog, and enabling the user to experience the interactive fun of wiping the glass on electronic devices such as mobile phones.

[0154] Similarly, in the examples illustratively shown in FIGS. 7c and 7d respectively, after the electronic device determines the current style, according to the user's action, in accordance with the handwriting generation methods corresponding to the sand painting style and the crayon style respectively, corresponding handwriting layers are generated, and after being subjected to mixing processing with the black semi-transparent mask, they are displayed, respectively displaying a heart-shaped graphic 12c with the sand painting style and a heart-shaped graphic 12d with the crayon style.

[0155] The examples shown in FIGS. 7a to 7d are all interaction examples in a partial display scenario. Referring to FIG. 7e, FIG. 7e shows an example in a full-screen display wallpaper scenario (full-screen dimmed state). In this example, the full-screen display wallpaper 121 is shown. After the user makes an action, the heart-shaped graphic 12e is displayed. In this scenario example, the screen of the electronic device is a folding screen. The interaction method provided by the embodiments of the present application can be applied to electronic devices with different forms of screens such as straight-plate screens or folding screens. Other examples are listed below.

[0156] Referring to FIGS. 7f to 7h, FIG. 7f shows an example in a scenario where the full-screen display wallpaper is dimmed. After the user draws a tick "√" shape on the screen, the corresponding tick shape is displayed. In this example, there is no longer an erasing effect similar to wiping the glass fog, and the position of the handwriting may not reveal the underlying wallpaper, but color filling is used. For example, white or other colors are used to fill the handwriting. Similarly, FIG. 7g shows that in the full-screen dimming scenario, after the user writes the character "王" on the screen, the interface 13b displays the corresponding handwriting, and the handwriting is filled with black.

[0157] FIG. 7h shows that in the screen-off scenario (i.e., the screen-off state), that is, when all the pixel points of the screen do not emit light, it is detected that the user makes an action of drawing a triangle on the screen, and the interface 13c displays a triangle consistent with the user's action. In this scenario, the electronic device may not support the AOD function or the user has not enabled the AOD function (i.e., the screen-off display is turned off), and the user selects to enable the screen-off interaction function.

[0158] The interaction method proposed by the embodiments of the present application supports the user to set the screen-off interaction, such as selecting the handwriting style and whether to enable the screen-off interaction. There are various implementation manners for the interface to implement the screen-off interaction setting. For example, in some embodiments, a control for implementing the screen-off interaction setting can be added to the screen-off display setting interface. For another example, in other embodiments, a separate screen-off interaction interface can be set, and the screen-off display interface and the screen-off interaction interface are juxtaposed as the next-level controls of "Desktop and Personalization".

[0159] For example, as shown in Figure 8, Figure 8 illustrates an example interface for implementing screen-off interaction settings. In the "Screen-off Display" settings interface 14a, controls for controlling screen-off interaction can be added, such as a control named "Screen-off Interaction" 14a1. Clicking control 14a1 opens the interface 14b for setting screen-off interaction. In interface 14b, users can choose to enable or disable the screen-off interaction function and select a handwriting style. For example, interface 14b displays four style options: ink wash, stylus, sand painting, and crayon. In practical applications, other styles can be added or existing styles can be removed as appropriate. The current style is determined based on the user's selection. If the user does not select a style after enabling the screen-off interaction function, the handwriting is generated and displayed according to the default style. The default style can be randomly selected from multiple styles or a style can be pre-specified.

[0160] Based on the above description, the interaction method proposed in this application will be further explained. This method can be applied to the electronic device 100 shown in Figure 1. For example, when the electronic device implements the interaction method proposed in this application, the hardware architecture shown in Figure 2 can be deployed in the electronic device, and / or, the software architecture shown in Figure 3 or Figure 4 can be adopted.

[0161] Please refer to Figure 9, which is a flowchart illustrating a specific embodiment of an interaction method provided in this application. The method may include, but is not limited to, the following steps:

[0162] S101: The electronic device detects that the current screen state is off.

[0163] When an electronic device detects that the current screen state is one of the following: screen off, partial display, or full screen dimming, it determines that the current screen state is screen off.

[0164] S102: The electronic device checks whether the screen-off interaction is enabled. If yes, proceed to S103; otherwise, end the process.

[0165] Electronic devices can provide a corresponding interactive interface. This interface displays controls for enabling or disabling the screen-off interaction function and shows multiple style options. Users can choose whether to enable the screen-off interaction function through the interface. When the screen-off interaction is enabled, users can select a style from the multiple styles available on the interface. If the system detects that the user has enabled the screen-off interaction function, it continues to search for information on the currently selected style.

[0166] For example, Figure 8 shows an example of the interactive interface for setting up the screen-off interaction function. The electronic device displays this interface and receives the user's selection, recording the style selected by the user. If the user enables the screen-off interaction function but does not select from multiple style candidates, the electronic device can randomly select a style as the selected style, or set the default style during factory settings, such as setting the default style to a stylus pen, and using the default style as the currently selected style.

[0167] It should be noted that in this embodiment, S101 can be executed before S102. In other embodiments, S102 can be executed before S101. That is, first check whether the screen-off interaction function is enabled. If it is not enabled, the process ends directly without executing S101.

[0168] The step numbers in the embodiments of this application are only used to distinguish different steps and are not intended to limit the order or timing of the execution of the steps.

[0169] S103: Electronic device queries information about the currently selected style.

[0170] S104: The electronic device senses whether the user is performing an action on the screen. If yes, proceed to S105; otherwise, continue waiting.

[0171] Electronic devices detect user actions on the screen, which can be achieved through a touch sensor (e.g., 180K). Once the touch sensor detects a user's touch on the touchscreen (the screen), it collects action data. This action data can include the coordinates and timestamps of multiple touch points collected at predetermined sampling periods. For example, when a user writes or draws on the screen, the coordinates of the user's finger or other body part touching the screen are collected every 8ms or 16ms, with each sampling point accompanied by a timestamp, thus obtaining the action data.

[0172] S105: The electronic device generates handwriting that matches the currently selected style and corresponds to the user's action.

[0173] Electronic devices generate handwriting to be displayed based on motion data collected by touch sensors and the currently selected style information. The handwriting to be displayed can be image data, containing the coordinate information of multiple pixels that make up the handwriting.

[0174] In this embodiment, each style corresponds to a specific handwriting generation algorithm to generate handwriting that matches the currently selected style. That is, the handwriting generation algorithm corresponding to the currently selected style is used to generate handwriting that matches the user's action. For example, APIs corresponding to the handwriting generation algorithms can be set, and by calling the APIs corresponding to various styles, handwriting corresponding to each style can be generated.

[0175] In this embodiment, generating handwriting corresponding to the user's action can be generating handwriting consistent with the user's action. For example, as shown in Figure 5 or Figures 7a to 7g, the displayed handwriting has a shape consistent with the user's action.

[0176] This embodiment uses handwriting as an example to illustrate the concept. In other embodiments, the content displayed based on the user's actions is not limited to handwriting, but can also be other content, such as displaying a dynamic wallpaper or static wallpaper corresponding to the user's actions.

[0177] S106: The electronic device displays handwriting corresponding to the actions performed by the user.

[0178] The following details how to generate handwriting with a specified style (i.e., the currently selected style) corresponding to the user's action in S105. As shown in Figure 10, action data containing 5 sampling points P1, P2, P3, P4, and P5 is first collected. The coordinate information and timestamp information of these sampling points are recorded: P1(t1), P2(t2), P3(t3), P4(t4), and P5(t5). Here, t1, t2, t3, t4, and t5 represent the timestamps corresponding to the 5 sampling points respectively. The difference between two adjacent timestamps is one sampling period, such as 8ms or 16ms. Based on the action data, Bezier curve fitting and interpolation are performed to obtain a Bezier curve. Points on the Bezier curve are sampled to obtain multiple sampling points. For example, B1 to B10 in the figure represent 10 sampling points obtained from sampling on the Bezier curve. Based on these 10 sampling points, brush elements of the corresponding style are added.

[0179] As shown in Figure 11, different styles correspond to different brush elements. The brush elements for the ink wash style can be brush strokes from a traditional Chinese calligraphy brush. The brush elements for the stylus style can be smooth lines.

[0180] Based on the positions of the sampling points on the Bezier curve, brush elements are evenly distributed across the curve. For example, a brush element is placed at sampling point B1, another at sampling point B2, and so on, resulting in a thick curve with added brush elements. Since different styles use different brush elements, the resulting thick curves exhibit different styles. Further processing of this thick curve, or leaving it unprocessed, yields the handwriting to be displayed.

[0181] There are several ways to add brush elements based on sampling points. For example, the sampling point can be added as the starting point of the stroke element, or as the center point of the stroke element. Alternatively, one or more brush elements can be added between two sampling points. This embodiment will not list them all.

[0182] Figure 10 uses a stylus style as an example. The brush element of the stylus can be a smooth line with uniform thickness. Therefore, after adding the corresponding stylus style brush element, you can get the handwriting example shown in Figure 10, which is filled with black.

[0183] The following details how to obtain Bezier curves based on motion data.

[0184] In this embodiment, a second-order Bezier curve is used, as shown in Figure 12. The user slides freely on the screen and draws a curve. The dashed line in Figure 12 represents the trajectory corresponding to the user's action. The touch sensor collects the user's action data. Every sampling period, it collects the coordinates and timestamp of the touch position. For example, it collects 5 sampling points P1-P5 as shown in Figure 10.

[0185] Connect points P1 and P2 with a straight line to obtain line segment P1P2 (represented by a dashed line in Figure 12). Take the midpoint of line segment P1P2 as d1, and then take the midpoint between line segment P1 and d1 as c1. Take P1 as the starting point S0 of the first Bezier curve, take d1 as the ending point S1 of the first Bezier curve, and take point c1 as the first control point. The value of t' corresponding to the Bezier curve changes from 0 to 1 to obtain the first Bezier curve.

[0186] Next, connect points P2 and P3 with a straight line to obtain line segment P2P3. Take the midpoint of line segment P2P3 as d2, take d1 as the starting point S1 of the second Bezier curve, take d2 as the ending point S2 of the second Bezier curve, take point P2 as the second control point c2, and change the value of t' from 0 to 1 to obtain the second Bezier curve.

[0187] Similarly, connect points P3 and P4 with a straight line to obtain line segment P3P4. Take the midpoint of the line segment as d3, take d2 as the starting point S2 of the second Bezier curve, take d3 as the ending point S3 of the second Bezier curve, and take P3 as the third control point c3. Change the value of t' from 0 to 1 to obtain the third Bezier curve. Continue in this manner, take the midpoint d4 of points P4 and P5, take d3 as the starting point S3, take d4 as the ending point S4, and take P4 as the control point c4 to obtain the fourth Bezier curve.

[0188] Connecting multiple Bezier curves yields a smooth Bezier curve as shown in Figure 12.

[0189] In other embodiments, a third-order Bezier curve (i.e., a 3rd-order Bezier curve) can be used. For example, a start point and an end point are determined, and two points between the start point and the end point are taken as control points to generate a third-order Bezier curve.

[0190] It should be noted that in some embodiments, after generating the Bezier curve, different styles can be further processed using the same or different methods when adding brush elements, or after adding brush elements.

[0191] For example, in this embodiment, as shown in Figure 13, after obtaining the Bezier curve, the Bezier curve is sampled. Based on the obtained multiple sampling points, brush elements of corresponding styles are added. For example, if the currently selected style is a stylus, then a stylus-style brush element is added to the Bezier curve; if the currently selected style is sand painting, then a stylus-style brush element is added to the Bezier curve; if the currently selected style is crayon, then a crayon-style brush element is added to the Bezier curve. For example, in this embodiment, these three styles of brush elements can be added evenly to obtain strokes with uniform line thickness.

[0192] In some embodiments, as shown in Figure 14, when generating sand painting style or crayon style handwriting, the brush elements can be randomly rotated and / or their positions jittered when adding sand painting style or crayon style brush elements.

[0193] As shown in Figure 15, random rotation can be achieved by rotating the original brush element at a random angle to obtain the rotated brush element. For example, the randomly rotated brush element can be placed at sampling point B2.

[0194] As shown in Figure 16, position jitter can be a random offset of the center point of the brush element relative to the sampling point by a predetermined amount. For example, the predetermined amount can be n pixels. In Figure 16, the center point of the brush element is offset by n pixels relative to the sampling point B2, where n can be 0-50.

[0195] Position jitter and random rotation can be combined or implemented separately. For example, position jitter and random rotation can be applied to the same brush element at the same time, or they can be applied to different brush elements. Different brush elements can be processed in a uniform way or in different ways. For example, brush elements on the same curve can be processed using random rotation instead of position jitter.

[0196] In this embodiment, the sand painting style is processed using a combination of random rotation and position jitter, while the crayon style is processed using position jitter.

[0197] In other embodiments, at least one of the processing methods such as random rotation and position jitter can be used to process brush elements of at least one of the above styles, without any specific limitation.

[0198] For example, in this embodiment, the ink wash style can be processed as shown in Figure 17. The ink wash style can be designed to meet writing needs, presenting an effect similar to writing with a brush. The ink wash style is suitable for writing Chinese characters and other languages. Therefore, the generated handwriting may include one or more strokes.

[0199] In the processing method corresponding to the ink painting style, the thickness of the strokes in the handwriting varies with speed; the faster the speed, the thinner the strokes, and the slower the speed, the thicker the strokes. This processing method requires further acquisition of the speed information when the user makes a movement. Speed ​​information can be determined based on the distance between adjacent sampling points and the sampling period in the action data. For example, the ratio of the distance between adjacent sampling points to the sampling period yields the speed. A higher speed value, meaning a faster speed, allows the corresponding stroke element to be reduced or lengthened, making the stroke line appear thinner relative to the original brush element. Conversely, a slower speed value, meaning a smaller speed, allows the corresponding stroke element to be thickened or enlarged, making the stroke line appear thicker relative to the original brush element.

[0200] It should be noted that in some embodiments, the speed data obtained based on the distance between adjacent sampling points and the sampling time interval (i.e., the sampling period) can be discrete data. In practical applications, the speed at a certain moment may undergo instantaneous changes, i.e., the speed change is drastic. If the line thickness (e.g., the thickness of a brush element) is adjusted based on the speed, abrupt changes in line thickness may occur, affecting the visual effect. The discrete speed data can be filtered to obtain a smoother speed curve. A low-pass filter can be used to filter the speed data, and the brush element at the corresponding position can be adjusted based on the filtered speed data. For example, in this embodiment, a one-euro filter is used as the low-pass filter. In other embodiments, the low-pass filter can be one or more combinations of a Lorentz low-pass filter, a Butterworth low-pass filter, a Kalman filter, and an RC low-pass filter.

[0201] In this embodiment or other embodiments, pressure information when the user makes an action can be further obtained. Based on the pressure information, the thickness of the strokes can be further adjusted so that the thickness of the strokes follows the pressure changes of the user's action. For example, the greater the pressure the user applies to the screen when making an action, the thicker the stroke at that location; the less pressure, the thinner the stroke at that location. Similarly, the stroke thickness can be adjusted by scaling the stroke element in a predetermined direction. For example, the predetermined direction can be along the length direction, along the width direction, or along a set angle direction. Scale-up along the length direction is stretching, scale-up along the width direction is thickening, and scale-up in both the length and width directions is proportional scaling; the same applies to shrinking.

[0202] For styles such as ink painting or handwriting, the placement direction of brush elements on the Bezier curve can be set to change according to the user's pen movement direction. The pen movement direction refers to the direction of movement when the user makes an action, such as the direction the user's finger slides on the screen. The placement direction of brush elements on the Bezier curve includes the placement direction of a single brush element or the splicing direction between adjacent brush elements. For example, as one implementation, when writing horizontally, a single brush element can be placed horizontally on the curve, and when writing vertically, a single brush element can be placed vertically on the curve. Another implementation is that the splicing direction between adjacent brush elements can be determined according to the pen movement direction. For example, taking the scenario of a user writing Chinese characters when the screen is off as an example, Chinese characters are divided into horizontal strokes from left to right and vertical strokes from top to bottom, as shown in Figure 18. When the user writes horizontally, two adjacent brush elements are spliced ​​horizontally; when the user writes vertically, two adjacent brush elements are spliced ​​vertically.

[0203] Furthermore, in this embodiment, brushstroke processing can also be used for ink painting style. Brushstroke processing refers to pre-processing at the beginning or end of a stroke. For example, in this embodiment, pre-processing can be to thicken the stroke elements by multiples at the beginning and / or end of the stroke. For instance, according to the writing habits of calligraphy, based on the stroke characteristics at the beginning of the stroke, the stroke elements at the beginning of the stroke are thickened by different multiples at different positions to obtain strokes that mimic the beginning of the stroke. The processing method at the end of the stroke is similar. In other embodiments, according to the writing habits of calligraphy, brush elements specifically for the beginning and end of the stroke can be added to the brush elements. When generating handwriting, the corresponding beginning or end brush elements are added at the beginning or end of the stroke.

[0204] It should be noted that, in other embodiments where the user selects the ink painting style, the handwriting can also be generated by recognizing the text corresponding to the user's action trajectory based on the user's actions and automatically generating characters in different fonts (such as Chinese characters). That is, the above-mentioned brush element splicing method is only one example. In some other embodiments, the brush element splicing method may not be used. Instead, after recognizing the text and other characters written by the user on the screen, a character in a certain font may be automatically displayed. The font can be selected by the user.

[0205] Based on the above exemplary description, the handwriting corresponding to the user's action can be obtained. As shown in Figure 19, in this embodiment, assuming that handwriting 00 is obtained after the above steps, a handwriting overlay 01 corresponding to the handwriting is then generated. As mentioned earlier, the full-screen darkening state can be achieved by overlaying an overlay on the wallpaper, for example, overlaying a black semi-transparent overlay. The overlay is a layer in which the pixel values ​​of the RGB three channels of each pixel are the same, and the value of the Alpha channel is also the same. The value of the Alpha channel (abbreviated as α) represents the transparency. The transparency can be from 0 to 1, or from 0 to 100. A transparency of 1 or 100 indicates complete opacity, and a transparency of 0 indicates complete transparency.

[0206] As shown in Figure 19, the full-screen overlay 02 is a semi-transparent overlay covering the wallpaper. In this embodiment, the RGB pixel values ​​of the handwriting overlay 01 generated based on the handwriting 00 are consistent with the RGB pixel values ​​of the full-screen overlay 02. Furthermore, the α value of each pixel in the handwriting overlay 01 is consistent with the α value of each pixel in the full-screen overlay 02, meaning their transparency is consistent.

[0207] Next, a subtractive blending operation is performed on the two layers, Handwriting Mask 01 and Full-Screen Mask 02. For example, the specific blending method could be an XOR operation, where pixels at the same position in both layers are XORed to obtain Full-Screen Mask 03. In Full-Screen Mask 03, the α value of each pixel in the area corresponding to the handwriting is 0.

[0208] The full-screen overlay 03 is blended with the wallpaper 04 (e.g., a full-screen wallpaper). For example, the blending mode can be multiply. The blended image data is then displayed on the screen as the data to be displayed. As shown in interface 15a in Figure 19, the visual effect is a darkened full-screen wallpaper, where the area where the user makes an action, i.e., the area corresponding to the handwriting, is transparent. The wallpaper pattern can be seen through the handwriting, presenting a visual effect simulating the erasing of fog from glass.

[0209] In this embodiment, after the handwriting is displayed, if no further action from the user is detected within a predetermined time period, the currently displayed handwriting can gradually fade away from the screen. As one possible implementation, the fading effect can be set from a spatial dimension, for example, fading from the edge to the center, from left to right, or from top to bottom; or, as another possible implementation, the fading effect can be set from a temporal dimension, for example, the transparency of each pixel on the handwriting gradually increases over time until it reaches the initial transparency (that is, the transparency is consistent with the full-screen overlay 02), then returns to a darkened full-screen state, and the handwriting is no longer visible. Alternatively, as yet another possible implementation, a combination of temporal and spatial dimensions can be used to present the fading effect.

[0210] Please refer to Figure 20. Figure 20 shows an example of a gradually disappearing display effect from the time dimension. At time t, interface 16a is an example of the interface currently displaying handwriting. At time t+1, the displayed interface is 16b. In interface 16b, the transparency of the area corresponding to the handwriting gradually increases over time, and the distinction between the handwriting and the overlay gradually weakens. At time t+2, as shown in interface 16c, the transparency of the pixels in the area corresponding to the handwriting is consistent with the transparency of the full-screen overlay 02, and the handwriting is completely invisible, returning to the original full-screen darkened state.

[0211] As shown in Figure 21, based on the above exemplary description, the interaction method provided in the embodiments of this application may include, but is not limited to, the following steps:

[0212] S201: The electronic device determines that the screen is in a screen-off state.

[0213] The screen-off state can be understood as the state of an electronic device between when the screen is turned off and when it is turned back on. In other words, any state between when the screen is turned off and when it is turned back on can be considered a screen-off state. To avoid confusion, it should be noted that a screen-off state can be one of these states. A screen-off state can also be a state of full-screen dimming, partial display, or any other state between when the screen is off and when it is turned back on.

[0214] Among them, "wake up the screen" refers to the user operating the electronic device to wake up the screen. The automatic display of AOD after the screen is turned off is different from "wake up the screen".

[0215] S202: Electronic devices detect the user's first action.

[0216] In some embodiments, the electronic device detects the user's first action by having the user perform an action on the screen, and the electronic device detects the user's action data through devices such as touch sensors and pressure sensors. In other embodiments, the user performs an action within the machine vision range of the electronic device, and the electronic device identifies the user's action by acquiring image data of the user's action.

[0217] S203: The electronic device responds to the first action and displays the first content; the first content corresponds to the first action.

[0218] The first content can be handwriting or other content. For example, in some embodiments, displaying the first content may be displaying the first handwriting corresponding to the first action. In other embodiments, the first content may be a dynamic wallpaper, a static wallpaper, or other specified information (such as weather, time, date, battery level, etc.).

[0219] The first content corresponds to the first action, which can be obtained by imitating the first action. For example, in some embodiments, displaying the first handwriting can mean displaying handwriting that imitates the first action. In other embodiments, the correspondence between the first content and the first action can also mean that there is a defined mapping relationship between the first content and the first action. For example, a specified touch action can be used as the first action, and a specified static wallpaper or live wallpaper can be used as the first content, establishing a defined mapping relationship between the first action and the first content. This mapping relationship can be customized by the user. In this way, when the user performs a specified touch action in the screen-off state, the desired live wallpaper or other content can be displayed, improving the user experience.

[0220] Based on the above exemplary description, it can be seen that in some embodiments, a background can be displayed before the first handwriting is displayed, and when the screen is off, the first handwriting can be displayed by displaying both the background and the first handwriting.

[0221] The background may include wallpaper and / or specified information. The wallpaper may be a full-screen wallpaper, a partial-screen live wallpaper, or a static wallpaper, etc. The specified information may include at least one of various messages such as time, weather, date, battery level, and notification messages.

[0222] For example, in the scene examples shown in Figure 5 or Figure 6b, the background includes the displayed time "10:49" and the date "July 12th, Saturday, August 3rd". As another example, in Figure 6c, the background includes a partial view of wallpaper 111 and the displayed time "10:49" and the date "July 12th, Saturday, August 3rd". In Figure 7a, the background includes a partial view of wallpaper 120. In Figures 7b to 7d, the background includes a partial view of wallpaper 121 and the displayed time "10:49" and the date "July 12th, Saturday, August 3rd". In Figure 7e, the background includes the full-screen wallpaper 121 and the displayed time information.

[0223] In some embodiments, the background and handwriting may be displayed together. For example, in interface 10c shown in FIG5, a background and heart-shaped handwriting are displayed; in FIG7a to FIG7e, a background containing wallpaper and heart-shaped handwriting are displayed. FIG7f and FIG7g show a full-screen display of wallpaper and handwriting.

[0224] In other embodiments, only the first stroke may be displayed without the background. For example, as shown in Figure 7h, in the example shown in Figure 7h, only the triangular stroke is displayed, and the background is not displayed; all other pixels on the display screen are off.

[0225] The first stroke displayed can be transparent or opaque. For example, in the examples shown in Figures 7a to 7e, the heart-shaped strokes are all transparent, and the area of ​​the wallpaper covered by the heart-shaped strokes in the background can be seen through the strokes. As another example, as shown in Figures 7f to 7h, the strokes are all opaque, and the underlying wallpaper cannot be seen through the strokes.

[0226] In embodiments where the background and handwriting are displayed together, the background includes a first wallpaper; when displayed, the first handwriting is shown on the first wallpaper. Exemplarily, the first handwriting is transparent; the area on the first wallpaper covered by the first handwriting is visible through the handwriting. For example, in FIG7a, the area of ​​wallpaper 120 covered by the handwriting "Li" can be seen through the handwriting; in the examples shown in FIG7b to 7e, the pattern of the area of ​​wallpaper covered by the handwriting can also be seen through the heart-shaped handwriting.

[0227] In some embodiments, a typical scenario is that a first wallpaper is displayed with a first transparency, for example, a full-screen wallpaper is darkened and covered with a mask; when the first handwriting is displayed on the first wallpaper, the first handwriting is displayed with a second transparency. The second transparency is different from the first transparency, for example, the second transparency is less than the first transparency. The transparency used to display the handwriting (second transparency) is less than the transparency used to display the wallpaper background (first transparency), which can present a visual effect that the area traversed by the handwriting is brightened. The transparency here refers to the value of the alpha channel (α Channel or Alpha Channel). For example, in the examples shown in Figures 7a to 7e, the area traversed by the handwriting is brighter than other darkened areas on the screen, presenting an effect of the part where the user's finger is swiped being brightened.

[0228] In other embodiments, to distinguish between wallpaper and handwriting, the first wallpaper can be displayed at a first brightness; after the user performs an action (first action), a first handwriting corresponding to the user's action can be displayed on the first wallpaper, with the first handwriting displayed at a second brightness, different from the first brightness. For example, in the interface 10c shown in FIG5, the brightness of the displayed heart-shaped handwriting can be different from the brightness of other areas on the screen. Or, for another example, in the interface examples shown in FIG7a to FIG7g, the brightness of the handwriting can be a first brightness, and the brightness of the wallpaper can be a second brightness, with the first brightness and the second brightness being different.

[0229] The first brightness and the second brightness can be achieved by directly controlling the brightness of the pixels. For example, in image data containing handwriting, the brightness value of the pixel corresponding to the handwriting is specified as the first brightness value, while the brightness value of the pixels in other areas of the screen, such as the area on the wallpaper other than the area displaying the handwriting, is set to the second brightness value.

[0230] In some embodiments, transparency or brightness can be controlled by overlaying or blending transparent overlays. For example, as illustrated in the specific embodiments listed above, as shown in FIG19, a first wallpaper displayed with a first transparency is obtained by overlaying a first overlay (e.g., full-screen overlay 02) on a first wallpaper (e.g., wallpaper 04), and each pixel in the first overlay has a first transparency α1; for example, the transparency of each pixel in the full-screen overlay 02 is α1.

[0231] The first handwriting (e.g., handwriting 06) displayed with a second transparency is obtained by overlaying a second overlay (e.g., full-screen overlay 03) onto the first wallpaper. The second overlay is a mixture of the first overlay and the first handwriting (which could refer to handwriting overlay 01 corresponding to handwriting 00). In the second overlay, the pixels corresponding to the first handwriting (handwriting 00) have a second transparency. For example, since the transparency of pixels in handwriting overlay 01 is also α1, when handwriting overlay 01 and full-screen overlay 02 are XORed, pixels with the same transparency have a transparency of 0, while pixels with different transparency remain at the first transparency. In the resulting full-screen overlay 03, the transparency of pixels corresponding to handwriting 00 becomes 0. When overlaid and mixed with wallpaper 04, a transparent or cutout effect is achieved in the area corresponding to the handwriting.

[0232] It should be noted that, in practical applications, as long as the transparency values ​​of each pixel in the area corresponding to the handwriting in the second mask layer are different from the transparency values ​​of pixels in other areas outside the handwriting in the mask layer, the handwriting can be displayed. The pixel values ​​corresponding to the handwriting do not have to be 0.

[0233] For example, in other embodiments, the process shown in FIG19 may not be performed. Instead, after determining the coordinates of the boundary pixels in the handwriting 00, the value of the α channel (i.e., the transparency) of each pixel at the position corresponding to the handwriting in the full-screen overlay 02 is modified to the second transparency α2, where α2 ≠ α1.

[0234] In other words, in the embodiment that uses a mask to control the first and second brightness, the second transparency corresponding to the handwriting in the mask is different from the first transparency corresponding to other areas in the wallpaper, so that the handwriting can be displayed and the displayed handwriting has a hollow effect.

[0235] In some embodiments, the interactive method proposed in this application supports displaying handwriting of a specified style. For example, Figure 7a shows handwriting in the style of ink painting, Figure 7b shows handwriting in the style of a stylus, Figure 7c shows handwriting in the style of sand painting, and Figure 7d shows handwriting in the style of crayon.

[0236] For example, displaying the first handwriting may be a first handwriting displaying variations in line thickness. Wherein, as described in the above embodiments, the line thickness can be determined based on the speed and / or pressure corresponding to the first action. As shown in Figure 7a, in the ink wash style, the line thickness varies with at least one of the speed and pressure when the user performs the action.

[0237] Exemplarily, displaying the first handwriting may be displaying the first handwriting with a constant line thickness. For example, as shown in FIGS. 7b to 7h, the line thickness of the displayed heart-shaped handwriting, or the displayed tick-shaped handwriting, or the displayed "king" character-shaped handwriting, or the displayed triangular handwriting is constant.

[0238] Exemplarily, displaying the first handwriting may be displaying the first handwriting including the first stroke, wherein the starting point and / or the ending point of the first stroke are displayed in a first manner; the middle part of the first stroke is displayed in a second manner, and the first manner is different from the second manner; wherein the middle part is the stroke between the starting point and the ending point of the first stroke.

[0239] For example, as shown in FIG. 7a, in the handwriting of the character "Li" displayed, the starting points and ending points of each stroke and the middle part of each stroke are displayed in different manners. Among them, the first stroke may be the "-" stroke. Taking the first stroke as an example, the display manner of the starting point and the ending point of this stroke is different from the display manner of the middle part of this stroke. The starting point and the ending point are specially processed. The starting point presents an obvious pen-stopping stroke when starting to write, and the ending point presents an obvious pen-stopping stroke when ending to write, while the middle part does not adopt the pen-stroke processing and does not adopt the pen-stroke display manner. Therefore, the starting point and the ending point are different from the display manner of the middle part.

[0240] In other embodiments, after recognizing characters such as text or symbols written by the user on the screen, characters of a specified font may be automatically displayed. The selection of the font may be set by the user. For example, when the user writes a Chinese character by hand on the screen, instead of displaying the Chinese character in the handwritten font imitating the user's writing on the screen, a Chinese character in the standard font of regular script or Song typeface or other fonts may be displayed.

[0241] In summary, the interaction method proposed in the embodiments of the present application can continuously draw the brush resource (brush element) along with the hand-slipping handwriting to obtain the handwriting, which may be a stroke or a graphic, etc. Perform a subtractive blending mode on the handwriting and the mask layer, and then superimpose it on an object such as a wallpaper displayed in the screen-off state, and display the obtained image data to form an effect of hollowing out the handwriting position. The displayed handwriting can be generated in real time following the sliding of the user's finger, enabling the user to experience the interactive fun of "wiping the glass". In addition, the embodiments of the present application can also select various styles and various brush effects, enabling the user to experience the "writing" fun of diverse styles.

[0242] Thus, the interaction method proposed in the embodiments of the present application can increase the interactive fun in the screen-off state and enhance the user experience. Moreover, after the finger is removed for a few seconds, the stroke or graphic effect gradually disappears and returns to the original darkened state, enriching the visual feedback.

[0243] The methods provided in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DWD), or a semiconductor medium (e.g., solid-state drive). (disk, SSD, etc.). The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An interaction method, characterized in that, Applied to electronic devices, the method includes: Determine that the screen of the electronic device is in an off state; Detect the user's first action; In response to the first action, in the screen-off state, the first content is displayed; the first content corresponds to the first action.

2. The method as described in claim 1, characterized in that, Display the first content, including: The first handwriting is displayed; the first handwriting corresponds to the first action.

3. The method as described in claim 2, characterized in that, Before displaying the first handwriting, the method further includes: In the screen-off state, a background is displayed; the background includes wallpaper and / or specified information. In the screen-off state, displaying the first handwriting includes: In the screen-off state, the background and the first handwriting are displayed.

4. The method as described in claim 3, characterized in that, The background includes the first wallpaper; The display of the background and the first handwriting includes: The first handwriting is displayed on the first wallpaper; wherein the first handwriting is transparent; and the area covered by the first handwriting on the first wallpaper is visible through the first handwriting.

5. The method as described in claim 3 or 4, characterized in that, The background includes the first wallpaper; The display background includes: The first wallpaper is displayed with the first level of transparency; The display of the background and the first handwriting includes: The first handwriting is displayed on the first wallpaper; wherein the first handwriting is displayed with a second transparency; the second transparency is different from the first transparency.

6. The method as described in claim 5, characterized in that, The second transparency is less than the first transparency.

7. The method according to any one of claims 2-6, characterized in that, The first handwriting is shown, including: Displays the first handwriting with the specified style.

8. The method according to any one of claims 2-7, characterized in that, The first handwriting is shown, including: The first handwriting displays variations in line thickness; wherein the line thickness is determined based on the speed and / or pressure corresponding to the first action.

9. The method according to any one of claims 2-7, characterized in that, The first handwriting is shown, including: Show the first stroke with a consistent line thickness.

10. The method according to any one of claims 2-8, characterized in that, The first handwriting is shown, including: The display includes a first stroke; wherein the starting point and / or ending point of the first stroke are displayed in a first manner; the middle part of the first stroke is displayed in a second manner, which is different from the first manner; wherein the middle part is the stroke between the starting point and the ending point of the first stroke.

11. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer programs; A processor for executing a computer program in the memory to implement the method as described in any one of claims 1-10.

12. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-10.

Images