A gesture interaction method, device and terminal equipment

Abstract

Embodiments of the present application provide a gesture interaction method and device, the method is applied to a terminal device with a folding screen, the folding screen of the terminal device comprises a first touch display area, a second touch display area and a third touch display area in a folded state; the third touch display area is between the first touch display area and the second touch display area; the method comprises determining that an angle value of an included angle formed by the first touch display area and the second touch display area is less than a set angle value; obtaining an input event acting on the third touch display area; in response to the input event, triggering the terminal device to execute an operation instruction corresponding to the input event. The method can respond to a gesture operation acting on the folding area of the folding screen of the terminal device, which is conducive to enriching the functions of the terminal device and improving the operation experience of the terminal device.

Description

Technical Field

[0001] This invention relates to the field of communications, and more particularly to a gesture interaction method, apparatus, and terminal device. Background Technology

[0002] With the diversification of terminal forms, various product forms such as foldable screens and irregular screens have entered the market. Traditional human-computer interaction methods can no longer meet user needs. For example, when a terminal device uses a foldable screen, there is an operable area in the folded area of ​​the folded screen. However, the operation of the folded area in the terminal device is limited in the current technology, and there is no function designed specifically for the folded area. Therefore, how to design the interaction method of the folded area of ​​the terminal device with a foldable screen to improve the user's sense of control and operation experience has become a problem to be solved. Summary of the Invention

[0003] This application provides a gesture interaction method, apparatus, and terminal device. The gesture interaction method is applied to a terminal device with a foldable screen. In its folded state, the foldable screen includes a first touch display area, a second touch display area, and a third touch display area, wherein the third touch display area is connected between the first and second touch display areas. This gesture interaction method can determine that the angle formed by the first and second touch display areas is less than a set angle value, acquire an input event acting on the third touch display area, and respond to the input event by triggering the terminal device to execute the operation command corresponding to the input event. This enriches the functionality of the terminal device and improves the user experience.

[0004] Firstly, this application provides a gesture interaction method applied to a terminal device with a foldable screen. When folded, the foldable screen includes a first touch display area, a second touch display area, and a third touch display area; the third touch display area is located between the first and second touch display areas. This method determines that if the angle between the first and second touch display areas is less than a set angle value, it indicates that the terminal is currently acquiring an input event acting on the third touch display area, and triggers the terminal device to execute the operation command corresponding to the input event. This method solves the problem of processing user input events when a touch response area exists in the folded area of ​​a foldable screen terminal device, thus improving the user experience.

[0005] In one possible design, the third touch display area includes the side area of ​​the terminal device.

[0006] In one possible design, the user's input event on the third touch display area is determined by detecting the user's gesture operation in the third touch display area. This design facilitates the determination of the user's gesture operation performed in the third touch display area.

[0007] In one possible design, the gestures input by the user on the third touch display area include one or more of the following: single-handed grip, two-handed grip, click, swipe, press, drag, and zoom.

[0008] In one possible design, the terminal device performs an application registration callback for the running application layer; in response to the input event, it determines the application operation instruction corresponding to the input event; and triggers the running application layer to execute the application operation instruction through a callback call or broadcast notification. This design ensures that the user's gesture operation is an operation on the application layer, enabling the application layer to respond to the user's gesture operation and perform the corresponding function.

[0009] In one possible design, the terminal device can distribute the input event to an application at the system layer; in response to the input event, the application at the system layer is triggered to execute the system operation command corresponding to the input event. This design allows it to determine that the user's gesture is an operation directed at the system, enabling the system to respond to the user's gesture and implement the corresponding system function.

[0010] In one possible design, the terminal device can acquire the characteristic parameters of the input event and trigger a task associated with the first touch display area and / or the second touch display area based on the characteristic parameters of the input event, executing the operation instruction corresponding to the input event. With this design, when the foldable screen of the terminal device is in a folded state, it is helpful to determine which touch display area the user's input event applied to the folded area is targeting.

[0011] In one possible design, the operation instructions include one or more of the following: taking a screenshot, adjusting the volume, turning pages, switching windows, opening or exiting the application, fast forwarding or rewinding.

[0012] Secondly, embodiments of this application provide a gesture interaction device that implements the gesture interaction method provided in the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function.

[0013] Thirdly, embodiments of this application provide a terminal device, which includes a processor and a memory; the memory is used to store a computer program, and the processor executes the computer program stored in the memory to cause the terminal device to perform the method in the first aspect or any possible implementation of the first aspect.

[0014] Fourthly, embodiments of this application provide a readable storage medium including a program or instructions that, when executed on a computer, cause the computer to perform the method of the first aspect or any possible implementation thereof.

[0015] The chip system mentioned above can be a system on chip (SOC) or a baseband chip, etc. The baseband chip can include processors, channel encoders, digital signal processors, modems and interface modules, etc. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0018] Figures 2a to 2c These are schematic diagrams illustrating different forms of terminal devices with foldable screens provided in the embodiments of this application;

[0019] Figure 3 This is a schematic diagram of a terminal device with an irregularly shaped screen provided in an embodiment of this application;

[0020] Figure 4a and Figure 4b This is an application scenario diagram of gesture interaction provided in the embodiments of this application;

[0021] Figure 5 This is a schematic diagram of the software system and hardware layer structure of a terminal device provided in an embodiment of this application;

[0022] Figure 6 This is a flowchart illustrating a gesture interaction method provided in an embodiment of this application;

[0023] Figure 7 This is a flowchart illustrating how an input event is converted into a gesture event, as provided in an embodiment of this application.

[0024] Figure 8This is a schematic diagram of another process for converting input events into gesture events, provided in an embodiment of this application.

[0025] Figure 9 This is a flowchart illustrating a functional operation corresponding to a gesture event, provided in an embodiment of this application.

[0026] Figure 10 This is a flowchart illustrating another functional operation corresponding to triggering a gesture event provided in an embodiment of this application;

[0027] Figure 11a This is an application diagram of a terminal device with a foldable screen provided in an embodiment of this application;

[0028] Figure 11b This is an application diagram of another terminal device with a foldable screen provided in an embodiment of this application;

[0029] Figure 12 This is a flowchart illustrating another gesture interaction method provided in an embodiment of this application;

[0030] Figure 13 This is a schematic diagram of the structure of a gesture interaction device provided in an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of another gesture interaction device provided in an embodiment of this application. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0033] The following describes embodiments of terminal devices and methods for implementing terminal devices. The terminal device can be a mobile phone (also known as a smart terminal device), a tablet personal computer, a personal digital assistant, an e-book reader, or a virtual reality interactive device, etc. The terminal device can access various types of communication systems, such as Long Term Evolution (LTE) systems, future 5th Generation (5G) systems, new radio access technology (NR), and future communication systems such as 6G systems; it can also be a wireless local area network (WLAN), etc.

[0034] For ease of explanation, the following embodiments will use a smart terminal device as an example.

[0035] This application provides a schematic diagram of the structure of a terminal device. Please refer to [link / reference]. Figure 1 The schematic diagram of the terminal device 100 shows that the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0036] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0037] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0038] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0039] The processor 110 may also include a memory for storing instructions and data. In one embodiment, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0040] In one embodiment, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0041] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0042] The I2S interface can be used for audio communication. In one embodiment, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0043] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0044] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback through Bluetooth headphones.

[0045] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In one embodiment, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to enable the display function of the terminal device 100.

[0046] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0047] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge terminal device 100, and can also be used for data transfer between terminal device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other terminal devices, such as AR devices.

[0048] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal device 100. In other embodiments of this application, the terminal device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0049] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In a wired charging embodiment, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In a wireless charging embodiment, the charging management module 140 receives wireless charging input via the wireless charging coil of the terminal device 100. While charging the battery 142, the charging management module 140 can also supply power to the terminal device via the power management module 141.

[0050] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In one embodiment, the power management module 141 can also be located within the processor 110. In another embodiment, the power management module 141 and the charging management module 140 can be housed in the same device.

[0051] The wireless communication function of the terminal device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0052] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0053] The mobile communication module 150 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0054] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0055] The wireless communication module 160 can provide solutions for wireless communication applications on the terminal device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0056] In one embodiment, antenna 1 of terminal device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling terminal device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0057] Terminal device 100 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0058] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel, which may specifically include a foldable screen, a non-circular screen, etc. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In one embodiment, the terminal device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0059] Terminal device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0060] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment, the ISP can be integrated into the camera 193.

[0061] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment, the terminal device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0062] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when terminal device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0063] Video codecs are used to compress or decompress digital video. Terminal device 100 may support one or more video codecs. Thus, terminal device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0064] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0065] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0066] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of terminal device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of terminal device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0067] Terminal device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0068] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0069] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.

[0070] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 100 answers a phone call or voice message, the receiver 170B can be brought close to the listener's ear to hear the voice.

[0071] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Terminal device 100 may be equipped with at least one microphone 170C. In some embodiments, terminal device 100 may be equipped with two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, terminal device 100 may be equipped with three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0072] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0073] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In one embodiment, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Terminal device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, terminal device 100 detects the intensity of the touch operation based on pressure sensor 180A. Terminal device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In one embodiment, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0074] The gyroscope sensor 180B can be used to determine the motion attitude of the terminal device 100. In one embodiment, the gyroscope sensor 180B can determine the angular velocity of the terminal device 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal device 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0075] The barometric pressure sensor 180C is used to measure air pressure. In one embodiment, the terminal device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0076] The magnetic sensor 180D includes a Hall effect sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In one embodiment, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0077] The 180E accelerometer can detect the magnitude of acceleration of the terminal device 100 in various directions (typically three axes). When the terminal device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the attitude of the terminal device, and can be applied to applications such as landscape / portrait switching and pedometers.

[0078] A distance sensor 180F is used to measure distance. The terminal device 100 can measure distance via infrared or laser. In one embodiment, during scene capture, the terminal device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0079] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outward through the LED. The terminal device 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 may use the proximity sensor 180G to detect when a user holds the terminal device 100 close to their ear for a call, so as to automatically turn off the screen to save power. The proximity sensor 180G can also be used in holster mode and pocket mode for automatic unlocking and screen locking.

[0080] The ambient light sensor 180L is used to sense the ambient light intensity. The terminal device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light intensity. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touches.

[0081] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0082] Temperature sensor 180J is used to detect temperature. In one embodiment, terminal device 100 uses the temperature detected by temperature sensor 180J to execute a temperature handling strategy. For example, when the temperature reported by temperature sensor 180J exceeds a threshold, terminal device 100 reduces the performance of the processor located near temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is below another threshold, terminal device 100 heats battery 142 to prevent abnormal shutdown of terminal device 100 due to low temperature. In still other embodiments, when the temperature is below yet another threshold, terminal device 100 boosts the output voltage of battery 142 to prevent abnormal shutdown due to low temperature.

[0083] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of terminal device 100, in a different position than display screen 194.

[0084] In one embodiment, the touch screen composed of touch sensor 180K and display screen 194 can be located in the side area or folded area of ​​terminal device 100, and is used to determine the location and gesture of the user's touch when the user's hand touches the touch screen; for example, when the user holds the terminal device, he can click any position on the touch screen with his thumb, then touch sensor 180K can detect the user's click operation and transmit the click operation to the processor, and the processor determines that the click operation is used to wake up the screen.

[0085] The bone conduction sensor 180M can acquire vibration signals. In one embodiment, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In one embodiment, the bone conduction sensor 180M can also be integrated into headphones to form bone conduction headphones. The audio module 170 can parse the voice signals based on the vibration signals from the vibrating bone segments of the vocal cords acquired by the bone conduction sensor 180M to realize voice functionality. The application processor can parse heart rate information based on the blood pressure signals acquired by the bone conduction sensor 180M to realize heart rate detection functionality.

[0086] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 100 can receive button input and generate key signal inputs related to user settings and function control of terminal device 100.

[0087] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0088] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0089] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 100. The terminal device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0090] When a terminal device uses a notched or foldable screen, its touch display can include multiple touch display areas. For example, a foldable screen in its folded state includes a folded area, which can also respond to touch. However, existing technologies have significant limitations in the operation of specific touch display areas, lacking dedicated operations for specific touch display areas. Therefore, this application provides a gesture interaction method. In this method, the side or folded area of ​​the terminal device has a touch response area. The terminal device can acquire input events from this touch response area and, in response to the input events, trigger the execution of the corresponding operation commands to achieve gesture operations on the side or folded area of ​​the terminal device, thereby improving the user experience.

[0091] Figures 2a to 2c The embodiments shown provide different forms of touch screen displays for terminal devices, wherein the touch screen display of the terminal device can be a foldable screen, such as... Figure 2a and Figure 2b As shown. The foldable screen 200 of this terminal device can be in two different states, including an unfolded state and a folded state. When the foldable screen 200 is in the unfolded state, as shown... Figure 2a As shown, the foldable screen 200, in its unfolded state, includes a touch display area 200A, within which users can input gestures. For example, a user can tap the icon of an application in the touch display area, and the terminal device, in response to the gesture, will display the corresponding application's user interface in the touch display area.

[0092] When the foldable screen 200 is in a folded state, such as Figure 2b As shown, the foldable screen 200, in its folded state, includes a first touch display area 200B, a second touch display area 200C, and a third touch display area 200D. The first touch display area 200B can be the side viewed by the user when using the terminal device; the second touch display area 200C can be the opposite side of the first touch display area 200B in the folded state; and the third touch display area 200D is the connecting surface of the foldable screen 200 in the folded state, connecting the first touch display area 200B and the second touch display area 200C. Figure 2b As shown. In the folded state, the angle formed by the first touch display area 200B and the second touch display area 200C is less than a set angle value. For example, when the angle formed by the first touch display area 200B and the second touch display area 200C is less than 60 degrees, the folded screen 200 is determined to be in a folded state. Optionally, when the folded screen 200 is fully folded, i.e., the angle formed by the first touch display area 200B and the second touch display area 200C is approximately 0 degrees, such as... Figure 2b The foldable screen on the right is shown. When the foldable screen 200 is in the folded state, the resulting third touch display area 200D can acquire user input events (such as user gestures) applied to that area. For example, if a user's finger slides up or down on the third touch display area, the terminal device can adjust the system volume in response to this gesture.

[0093] Optionally, the third touch display area 200D of the foldable screen 200 may also include the side area of ​​the terminal device. For example, when the foldable screen 200 is in the unfolded state, the third touch display area 200D located on the side of the terminal device is as follows: Figure 2c As shown, the third touch display area is Figure 2c The area enclosed by the dotted line in the diagram is the third touch display area 200D, which can acquire user input events (such as user gestures) applied to this area. It is understood that... Figure 2c The side area of ​​the terminal device shown is an example. The third touch display area located on the side of the terminal device may also include other side areas. This embodiment does not impose any limitations.

[0094] Optionally, the touch screen of the terminal device can also be an irregularly shaped screen, such as... Figure 3As shown, the irregularly shaped screen includes a first touch display area 300A of the terminal device, second touch display areas 300B on both sides, and a third touch display area 300C. The first touch display area 300A is the side viewed by the user when using the terminal device, while the second touch display areas 300B and third touch display areas 300C are the sides of the terminal device held by the user when viewing the first touch display area 300A. Figure 3 As shown. The second touch display area 300B and the third touch display area 300C can acquire user input events (such as user gestures) applied to these areas. For example, when a user holds the terminal device with one hand, the user's thumb can tap the second touch display area 300B twice consecutively. The terminal device responds to this gesture by taking a screenshot of the first touch display area 300A. It is understood that the second touch display area 300B and the third touch display area 300C can be flat or curved surfaces extending laterally from the first touch display area 300A.

[0095] Based on the above description of the different forms of touch screen displays for terminal devices. Figure 4a and Figure 4b The illustrated embodiment provides an application scenario for gesture interaction using a user-held terminal device. The gesture interaction methods described in this embodiment include gripping methods and gesture triggering methods. Gripping methods refer to how the user holds the terminal device, which can include single-handed gripping and two-handed gripping. For example, when a user holds the terminal device with one hand, they trigger a gesture operation by touching the response area of ​​the terminal device's touch panel (TP) with their finger. Gesture triggering methods refer to the gesture operation performed by the user on the touch display screen, which may include, but is not limited to, swiping, pressing, clicking, dragging, and zooming operations. The parameters of a swiping operation may include, but are not limited to, the number of fingers used, the distance swiped, and the speed of swiping. For example, a gesture operation triggered by the user in the TP response area involves the user using their finger to... Figure 2b The third touch area shown is 200D, which slides 10 mm. Parameters for the pressing operation may include, but are not limited to, the number of fingers used during the press, the pressure applied by the fingers, etc. For example, when the user presses firmly... Figure 2b When the third touch display area 200D is shown, this touch display area can detect that the user's gesture operation is a press operation. The parameters of the click operation can include, but are not limited to, the number of clicks, the click speed, etc. For example, when the user clicks multiple times consecutively within a preset time period... Figure 2b When the third touch display area 200D is shown, this touch display area can detect that the user's gesture operation is a continuous click operation. The parameters of the drag operation can include, but are not limited to, the drag distance and the drag speed. For example, the user can drag... Figure 2a The icons within the first touch display area 200A shown can be repositioned within that area. Parameters for the zoom operation may include, but are not limited to, the number of fingers used for zooming, the zoom range, etc. For example, a user can zoom in on... Figure 2a The first touch display area 200A shown can be zoomed in and out using the thumb and forefinger.

[0096] Based on the above description of gesture interaction methods, the following will use... Figure 2b Taking foldable screens as an example, this section introduces scenarios where users interact with gestures on their handheld devices, such as... Figure 4a As shown. Figure 4a This describes an application scenario where a user inputs gestures to the third touch display area 200D when the foldable screen is in its folded state. In this scenario, the first touch display area 200B faces the user, the second touch display area 200C is located behind the first touch display area 200B in the folded state, and the third touch display area 200D connects the first touch display area 200B and the second touch display area 200C. Figure 4a As shown. The third touch display area 200D has a TP response area, so the third touch display area 200D can receive the user's gesture operation on this area. For example, the user's gesture operation on the third touch display area 200D is to slide 10 mm to the right within the TP response area of ​​the third touch display area 200D. Figure 4a As shown.

[0097] The following is based on Figure 3 Taking a non-circular screen as an example, this section introduces scenarios for gesture interaction on user gesture terminal devices, such as... Figure 4b As shown. Figure 4b In scenarios where a user holds the terminal device with one hand (right hand), the second touch display area 300B and the third touch display area 300C of the irregularly shaped screen have TP response areas. Therefore, the system can detect that the user's gesture interaction with the terminal device in this scenario is a one-handed grip. For example, if the user holds the terminal device with their right hand and slides their thumb upwards within the TP response area of ​​the second touch display area 300B, then the second touch display area 300B can detect through the TP response area that the user's gesture interaction with the terminal device in this scenario is a one-handed grip and an upward slide within the second touch display area 300B.

[0098] The following is combined with Figure 4a and Figure 4b The scenario shown illustrates a user's handheld terminal device interacting with gestures, exemplifying the structure of the terminal device's software system and hardware layer. Please refer to [link to relevant documentation]. Figure 5The terminal device software system and hardware layer mainly consist of three modules: the hardware layer, kernel space, and user space. The hardware layer generates corresponding hardware interrupt signals based on user operations and may include, but is not limited to, touchscreen displays and sensors. The kernel space receives and reports hardware interrupt information generated by the hardware layer, generates input events based on the hardware interrupt information, and uploads the input events to the user space. The kernel space may include multiple drivers, such as TP drivers and sensor drivers. The user space is used for reading, processing, and distributing input events and includes device nodes and application framework layers. The device node serves as the hub connecting the kernel space and user space. The following describes the hardware and software workflow for gesture interaction on a user-held terminal device, which may include the following steps:

[0099] When the TP response area in the hardware layer touch display receives a user's gesture operation, a corresponding hardware interrupt signal is generated and sent to the kernel space;

[0100] After receiving the hardware interrupt signal, the kernel space can process the hardware interrupt signal into an input event, which includes information such as touch coordinates and the timestamp of the touch operation, and then report the input event to the user space.

[0101] Device nodes in user space can acquire the input events and then process them through the application framework layer to respond to user input gestures.

[0102] Among them, with Figure 4a Taking a foldable screen as an example, when a user inputs a gesture operation on the third touch display area, such as pressing and holding for two seconds and then swiping down, the hardware layer will generate a corresponding hardware interrupt signal and send it to the kernel space. Upon receiving the hardware interrupt signal, the kernel space processes it into an input event. This input event includes the third touch display area as the triggering area for the gesture operation, and the gesture type being either a press or a swipe. After the user space receives the input event, it calls the application framework layer. If it detects that no application is currently running on the terminal device, the user space will trigger a system action: unlocking the screen. The terminal device, in response to the input event, will unlock the screen.

[0103] The following will describe in detail a gesture interaction method provided by an embodiment of this application. Please refer to [link to relevant documentation]. Figure 6 This gesture interaction method can be applied to terminal devices with foldable or irregularly shaped screens, and may specifically include the following steps:

[0104] S601, determine that the angle value of the angle formed by the first touch display area and the second touch display area is less than the set angle value.

[0105] When the touch screen of the terminal device is a foldable screen, the foldable screen of the terminal device includes a first touch display area, a second touch display area, and a third touch display area in the folded state, wherein the third touch display area is located between the first touch display area and the second touch display area, for example, Figure 2b In the folded state of the terminal device shown, the third touch display area 200D is located between the first touch display area 200B and the second touch display area 200C. Specifically, the folded state of the terminal device's screen can be determined when the angle formed by the first and second touch display areas is less than a set angle value. For example, when the angle between the first and second touch display areas is less than 60 degrees, the folded state of the terminal device's screen can be determined. It is understood that the aforementioned third touch display area is the folded area generated by the terminal device's folded screen in the folded state, and this folded area can acquire user input events. Optionally, when the terminal device's touch display screen is an irregularly shaped screen, the third touch display area of ​​the terminal device can be a side area of ​​the irregularly shaped screen, for example... Figure 3 The terminal device shown has touch display areas 300B and / or touch display areas 300C on both sides.

[0106] S602, acquire the input event acting on the third touch display area.

[0107] The third touch display area includes a TP (touch response) area, which is used to detect user gestures input within the third touch display area. These gestures may include, but are not limited to, swiping, pressing, clicking, dragging, and zooming. For example, Figure 4a The third touch display area 200D shown can be used to detect user input gestures (such as swipe operations, press operations, etc.). These user input gestures can be acquired by a touch sensor. For example, the TP response area triggers the touch sensor in the hardware layer to acquire the user input gesture, and the hardware layer generates a corresponding hardware interrupt signal based on the acquired gesture.

[0108] Optionally, after the hardware layer generates a corresponding hardware interrupt signal, it can transmit the hardware interrupt signal to the kernel space. After the kernel space receives the hardware interrupt signal, it can determine the user's input event on the third touch display area based on the gesture operation corresponding to the hardware interrupt signal. The input event includes parameters such as event type, event trigger time, and operation data. For example, when the TP response area detects that the user is performing a gesture operation on the third touch display area, assuming that the gesture operation is the user's finger on... Figure 4a If a designated area in the third touch display area 200D of the foldable screen is slid 10 mm to the right, the touch sensor will generate a corresponding hardware interrupt signal and transmit it to the kernel space. The kernel space will then generate an input event based on the hardware interrupt signal. This input event is a swipe operation with a trigger time of 1 second and the operation data being a 10 mm swipe to the right.

[0109] Optionally, when the kernel space receives an input event, it needs to encapsulate the input event. This encapsulation can be achieved by the kernel space directly converting the input event into a gesture event, or by reporting the input event to the user space, where the user space then converts the input event into a gesture event. A gesture event is an event that can be read and processed by either the kernel space or the user space, including a gesture operation and the object responding to that gesture operation. In one feasible implementation, the kernel space can directly convert the input event into a gesture event. The software modules involved include both the kernel space and the user space, such as... Figure 7 As shown, the specific steps may include:

[0110] s11, The kernel space uses an algorithm to identify input events and convert the input events into gesture events;

[0111] s12, the kernel space reports the gesture event to the user space.

[0112] When the TP driver in the kernel space receives an input event, it inputs the input event to the algorithm recognition module in the kernel space. The algorithm recognition module can use algorithms (such as matching algorithms, neural network-based algorithms, etc.) to recognize the input event, determine the event type, event trigger time, touch data, etc., and convert the input event into a gesture event. For example, when the input event includes an event type of swipe trigger, an event trigger time of 1 second, and touch data of swiping upwards by 10 millimeters, the kernel space uses an algorithm to recognize the input event and converts it into a gesture event. The gesture event includes a gesture operation in which the user's finger swipes upwards by 10 millimeters in the touch response area, and the object responding to the gesture operation is the operating system.

[0113] In another feasible implementation, the kernel space can report the input event to the user space, which then converts the input event into a gesture event. The software modules involved include both the kernel space and the user space, such as... Figure 8 As shown, the specific steps may include:

[0114] s21, the kernel space reports the input event to user space;

[0115] s22, The user space uses an algorithm to identify the input event and convert the input event into a gesture event.

[0116] In this process, after receiving an input event, the kernel space does not perform any operations on the input event but directly reports it to the user space. The application framework layer in the user space can use an algorithm to identify the input event, determine its event type, trigger time, and touch data, and convert it into a gesture event. The gesture event includes a gesture operation and the object responding to that gesture operation. For example, when the input event includes an event type of click trigger, an event trigger time of 1 second, and touch data of two consecutive clicks, the user space uses an algorithm to identify the input event and convert it into a gesture event. The gesture event includes the gesture operation of the user's finger clicking twice consecutively in the touch response area, and the object responding to that gesture operation is the operating system. It is understandable that this differs from... Figure 7 In the illustrated embodiment, the kernel space uses an algorithm to identify input events and convert them into gesture events. Figure 8 In the illustrated embodiment, the user space uses an algorithm to identify input events and converts the input events into gesture events. The processing flow of the two embodiments is different, but the result is that the user space determines the gesture event so as to further respond to the gesture event to perform the corresponding function operation.

[0117] S603, in response to the input event, the terminal device is triggered to execute the operation instruction corresponding to the input event.

[0118] After the terminal device receives an input event applied to the third touch display area, it can trigger the terminal device to execute a corresponding operation command. The terminal device can execute the corresponding operation command based on parameters such as the event type, event trigger time, and operation data of the input event. The operation command may include, but is not limited to, operations such as screenshotting, volume adjustment, page turning, window switching, opening or exiting an application, fast forwarding or rewinding, etc. For example, when the input event is a double-tap operation, the corresponding operation command is a screenshot. Since the operation command corresponding to the input event can be a system operation command for a system-level application or an application operation command for an application-level application, if the terminal device is currently running an application, S603 may specifically include the following steps:

[0119] Perform application registration callbacks for running application-layer applications;

[0120] In response to the input event, determine the application operation instruction corresponding to the input event;

[0121] The application in the running application layer is triggered to execute the application operation instructions through callback calls or broadcast notifications.

[0122] If the terminal device does not currently have any applications running, S603 may specifically include the following steps:

[0123] Distribute the input events to applications at the system layer;

[0124] In response to the input event, the system layer application is triggered to execute the system operation command corresponding to the input event.

[0125] When a terminal device has a running application, the input event can be determined to be an operation targeting that application. The corresponding application operation instruction is then determined based on the input event, and the running application is triggered to execute the corresponding instruction. For example, if the input event is a double-click operation, and the currently running application is a video playback application, the corresponding application operation instruction is a video screenshot. The terminal device will then trigger the video playback application to perform a screenshot operation to obtain a screenshot of the currently playing video. When no application is running on the terminal device, the input event can be determined to be a system operation instruction corresponding to a system-level application. The system-level application is then triggered to execute the corresponding system operation instruction based on the input event. For example, if the input event is a two-second long press followed by a 10mm slide along the long side of the third touch display area, and the terminal device detects that no application is currently running, the corresponding system operation instruction is to unlock the screen. The terminal device's system will then unlock the screen to display the user interface.

[0126] Specifically, the user space in the terminal device's software system can distribute gesture events converted from input events to trigger corresponding functional operations. The distribution of gesture events can directly trigger system behavior based on the type and information of the gesture event, or trigger custom behavior of the application through application registration callbacks. In one feasible implementation, when the user space responds to a gesture event and triggers system behavior, the software modules involved include the user space, such as... Figure 9 As shown, the following steps may be included:

[0127] s31, The application framework layer dispatches the gesture event;

[0128] s32, The application framework layer triggers system behavior based on the type and information of the gesture event.

[0129] The event manager manages gesture events. System behaviors are directly dispatched by the application framework layer to execute corresponding functional operations. For example, if the gesture event is to press hard and then click once, the corresponding functional operation is to switch windows; or if the gesture event is to slowly slide down 10 millimeters, the corresponding functional operation is to lower the volume.

[0130] In another feasible implementation, when the user space responds to the gesture event and triggers a custom behavior of the application through an application registration callback, the software modules involved include the user space and the application layer, such as... Figure 10 As shown, the following steps may be included:

[0131] s41, The application layer uses an application registration callback mechanism to add the application to the application layer;

[0132] s42, The application framework layer notifies the application to trigger application behavior;

[0133] s43, the application in the application layer performs the corresponding functional operation.

[0134] The application registration callback mechanism first submits registration information to the event manager of the application architecture layer. Then, when the application framework layer responds to a gesture event and triggers application behavior, it notifies the application through callback calls, broadcasts, etc., to trigger the application's custom behavior. For example, when the gesture event is pressing down with a finger for 2 seconds and then sliding down 10 millimeters, the user space will recognize this gesture operation as triggering the camera application. Then, by calling the interface of the application framework layer, the camera application is launched, and then the camera driver is launched by calling the kernel space to capture images or videos through the camera.

[0135] Optionally, when the terminal device uses a foldable screen and the foldable screen is in a folded state, since there are a first touch display area and a second touch display area, when an input event acting on the third touch display area is detected, it is necessary to determine which touch display area the operation instruction corresponding to the input event is for. S603 may specifically include the following steps:

[0136] Obtain the feature parameters of the input event, the feature parameters including one or more of the following: pressure, sliding direction, application location, and number of touches;

[0137] Based on the characteristic parameters of the input event, a task associated with the first touch display area and / or the second touch display area is triggered to execute the operation instruction corresponding to the input event.

[0138] When an input event is detected acting on the third touch display area, the terminal device can acquire the characteristic parameters of the input event. These characteristic parameters indicate which specific touch display area the user is acting on when performing a gesture operation within the third touch display area. The characteristic parameters may include, but are not limited to, pressure intensity, swipe direction, position, and number of touches. For example, the terminal device can pre-set an input event at a first position within the third touch display area to trigger the execution of a corresponding operation command for a task associated with that first touch display area, and an input event at a second position within the third touch display area to trigger the execution of a corresponding operation command for a task associated with that first touch display area. Figure 11a As shown. For example, the terminal device can be pre-configured so that when the user faces the first touch display area, swiping right on the third touch display area triggers the execution of a corresponding operation command for the task associated with the first touch display area, and swiping left on the third touch display area triggers the execution of a corresponding operation command for the task associated with the second touch display area, such as... Figure 11b As shown. Based on the characteristic parameters of the input event, the target touch display area can be determined, thereby triggering the terminal device to execute the operation command corresponding to the input event for the task associated with the target touch display area. The task associated with the target touch display area includes applications that are running and displayed in the target touch display area; for example, the task associated with the target touch display area could be a video application playing a video.

[0139] For example, the task associated with the first touch display area of ​​the current terminal device is that a music application is playing music, and the task associated with the second touch display area is that the user is using the WeChat application to communicate instantly with other WeChat users. Suppose that while using WeChat, the user receives a video call request from another WeChat user and needs to lower the volume of the music being played so that the call is not disturbed. The user can then input a gesture at the first position of the third touch display area, for example, swiping 10 millimeters to the left within the first position. Based on the input event and its characteristic parameters, the terminal device determines the target touch display area as the first touch display area and triggers the terminal device to lower the volume of the music being played in the music application.

[0140] As can be seen, the embodiments of this application provide a gesture interaction method. The side area or folded area of ​​the terminal device in the gesture interaction method has a touch response area. By acquiring the input event of the touch response area and converting the input event into a gesture event, and then triggering the corresponding operation according to the gesture event, gesture operation on the side area or folded area of ​​the terminal device can be realized, thereby improving the operation experience of the terminal device.

[0141] Combination Figure 6 The description in the illustrated embodiments, Figure 12 The illustrated embodiment provides a flowchart of a gesture interaction method for operating a terminal device. Taking a phone with a non-standard screen as an example, when a user holds the phone with one hand, the processor controls the touch response area to sense the gesture. The processor performs gesture recognition on the touch response area to determine the user's grip and gesture triggering method on the side screen of the phone. For example, the user holds the phone with one hand and slides their finger up and down on the side screen. After determining the user's gesture operation on the side screen, the processor detects the currently running application. For example, if the currently running application is a camera application, the processor associates the user's gesture operation with the current application. If the predefined operation corresponding to the gesture event is determined to be adjusting the focus, the processor calls the camera application to perform the operation, adjusting the focus of the current camera lens to achieve the focus adjustment function.

[0142] This application provides a gesture interaction device, such as... Figure 13 As shown. The gesture interaction device 1300 can be used to perform... Figure 6 The gesture interaction method includes:

[0143] The determining unit 1301 is used to determine that the angle value of the included angle formed by the first touch display area and the second touch display area is less than a set angle value;

[0144] Acquisition unit 1302 is used to acquire input events acting on the third touch display area;

[0145] The processing unit 1303 is configured to respond to the input event and trigger the terminal device to execute the operation instruction corresponding to the input event.

[0146] In one implementation, the third touch display area includes the side area of ​​the terminal device.

[0147] In one implementation, the acquisition unit 1302 can specifically be used for:

[0148] Detect the gesture operation input by the user in the third touch display area;

[0149] The user's input event applied to the third touch display area is determined based on the gesture operation.

[0150] In one implementation, the gesture operation includes one or more of the following: single-handed grip, two-handed grip, click operation, swipe operation, press operation, drag operation, and zoom operation.

[0151] In one implementation, the processing unit 1303 can specifically be used for:

[0152] Perform application registration callbacks for running application-layer applications;

[0153] In response to the input event, determine the application operation instruction corresponding to the input event;

[0154] The application in the running application layer is triggered to execute the application operation instructions through callback calls or broadcast notifications.

[0155] In one implementation, the processing unit 1303 can specifically be used for:

[0156] Distribute the input events to applications at the system layer;

[0157] In response to the input event, the system layer application is triggered to execute the system operation command corresponding to the input event.

[0158] In one implementation, the processing unit 1303 is further configured to:

[0159] Obtain the feature parameters of the input event, the feature parameters including one or more of the following: pressure, sliding direction, application location, and number of touches;

[0160] Based on the characteristic parameters of the input event, a task associated with the first touch display area and / or the second touch display area is triggered to execute the operation instruction corresponding to the input event.

[0161] In one implementation, the operation instructions include one or more of the following: taking a screenshot, adjusting the volume, turning pages, switching windows, opening or exiting an application, fast forwarding or rewinding.

[0162] This application provides another gesture interaction device, such as... Figure 14 As shown. The gesture interaction device 1400 may include a processor 1401, wherein, Figure 13 The functions implemented by the determining unit 1301, the acquiring unit 1302, and the processing unit 1303 shown can all be implemented by the processor 1401. The processor 1401 may include one or more processors, such as one or more central processing units (CPUs), network processors (NPs), hardware chips, or any combination thereof. If the processor 1401 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0163] The gesture interaction device 1400 may also include a memory 1402 for storing program code, etc. The memory 1402 may include volatile memory, such as random access memory (RAM); the memory 1402 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 1402 may also include a combination of the above types of memory.

[0164] The processor 1401 and memory 1402 described above can be used to implement Figure 6 In the gesture interaction method described above, the processor 1401 is used to determine that the angle value of the angle formed by the first touch display area and the second touch display area is less than a set angle value;

[0165] Acquire input events that act on the third touch display area;

[0166] In response to the input event, the terminal device is triggered to execute the operation instruction corresponding to the input event.

[0167] In one implementation, the third touch display area includes the side area of ​​the terminal device.

[0168] In one implementation, processor 1401 can specifically be used for:

[0169] Detect the gesture operation input by the user in the third touch display area;

[0170] The user's input event applied to the third touch display area is determined based on the gesture operation.

[0171] In one implementation, the gesture operation includes one or more of the following: single-handed grip, two-handed grip, click operation, swipe operation, press operation, drag operation, and zoom operation.

[0172] In one implementation, processor 1401 can specifically be used for:

[0173] Perform application registration callbacks for running application-layer applications;

[0174] In response to the input event, determine the application operation instruction corresponding to the input event;

[0175] The application in the running application layer is triggered to execute the application operation instructions through callback calls or broadcast notifications.

[0176] In one implementation, processor 1401 can specifically be used for:

[0177] Distribute the input events to applications at the system layer;

[0178] In response to the input event, the system layer application is triggered to execute the system operation instruction corresponding to the input event. In one implementation, the processor 1401 may specifically be used to:

[0179] Obtain the feature parameters of the input event, the feature parameters including one or more of the following: pressure, sliding direction, application location, and number of touches;

[0180] Based on the characteristic parameters of the input event, a task associated with the first touch display area and / or the second touch display area is triggered to execute the operation instruction corresponding to the input event.

[0181] In one implementation, the operation instructions include one or more of the following: taking a screenshot, adjusting the volume, turning pages, switching windows, opening or exiting an application, fast forwarding or rewinding.

[0182] It should be noted that the device in the above embodiments can be a terminal device, or a chip or other combination device or component with the above-mentioned terminal functions applied in the terminal device.

[0183] This application also provides a readable storage medium, which includes a program or instructions that, when run on a computer, cause the computer to execute the gesture interaction method performed by the gesture interaction device in the above method embodiments.

[0184] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0185] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0186] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A gesture interaction method, applied to a terminal device with a foldable screen, characterized in that, The foldable screen of the terminal device, in its folded state, includes a first touch display area, a second touch display area, and a third touch display area; the third touch display area is located between the first touch display area and the second touch display area; the method includes: If the angle between the first touch display area and the second touch display area is less than a set angle value, the foldable screen is in a folded state. When the foldable screen is in a folded state, an input event acting on the third touch display area is acquired; The feature parameters of the input event are obtained, and the feature parameters are used to indicate that the gesture operation input by the third touch display area is applied to the first touch display area or the second touch display area; Based on the characteristic parameters of the input event, the operation instruction corresponding to the input event is executed for the task associated with the first touch display area or the second touch display area.

2. The method according to claim 1, characterized in that, The third touch display area includes the side area of ​​the terminal device.

3. The method according to claim 1 or 2, characterized in that, The acquisition of input events acting on the third touch display area includes: Detect the gesture operation input by the user in the third touch display area; The user's input event applied to the third touch display area is determined based on the gesture operation.

4. The method according to claim 3, characterized in that, The gesture operations include one or more of the following: single-hand grip, two-hand grip, click operation, swipe operation, press operation, drag operation, and zoom operation.

5. A gesture interaction device, characterized in that, The gesture interaction device includes a foldable screen, which, in its folded state, includes a first touch display area, a second touch display area, and a third touch display area; the third touch display area is located between the first touch display area and the second touch display area. The gesture interaction device also includes: The determining unit is used to determine that the angle formed by the first touch display area and the second touch display area is less than a set angle value; when the angle formed by the first touch display area and the second touch display area is less than the set angle value, the foldable screen is in a folded state; The acquisition unit is used to acquire input events applied to the third touch display area when the foldable screen is in a folded state. The acquisition unit is further configured to acquire feature parameters of the input event, the feature parameters being used to indicate that the gesture operation input by the third touch display area acts on the first touch display area or the second touch display area; The processing unit is configured to execute the operation instruction corresponding to the input event for the task associated with the first touch display area or the second touch display area, based on the characteristic parameters of the input event.

6. The apparatus according to claim 5, characterized in that, The third touch display area includes the side area of ​​the gesture interaction device.

7. The apparatus according to claim 5 or 6, characterized in that, When the acquisition unit acquires an input event acting on the third touch display area, it is specifically used for: Detect the gesture operation input by the user in the third touch display area; The user's input event applied to the third touch display area is determined based on the gesture operation.

8. The apparatus according to claim 7, characterized in that, The gesture operations include one or more of the following: single-hand grip, two-hand grip, click operation, swipe operation, press operation, drag operation, and zoom operation.

9. A terminal device, characterized in that, The terminal device includes a foldable screen, which, in its folded state, includes a first touch display area, a second touch display area, and a third touch display area; the third touch display area is located between the first touch display area and the second touch display area. The terminal device also includes: a memory and a processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory, so that the terminal device performs the following method: If the angle between the first touch display area and the second touch display area is less than a set angle value, the foldable screen is in a folded state. When the foldable screen is in a folded state, an input event acting on the third touch display area is acquired; The feature parameters of the input event are obtained, and the feature parameters are used to indicate that the gesture operation input by the third touch display area is applied to the first touch display area or the second touch display area; Based on the characteristic parameters of the input event, the operation instruction corresponding to the input event is executed for the task associated with the first touch display area or the second touch display area.

10. The device according to claim 9, characterized in that, The third touch display area includes the side area of ​​the terminal device.

11. The device according to claim 9 or 10, characterized in that, When the processor acquires an input event acting on the third touch display area, it is specifically used for: Detect the gesture operation input by the user in the third touch display area; The user's input event applied to the third touch display area is determined based on the gesture operation.

12. The device according to claim 11, characterized in that, The gesture operations include one or more of the following: single-hand grip, two-hand grip, click operation, swipe operation, press operation, drag operation, and zoom operation.

13. A readable storage medium, characterized in that, Includes a program or instructions that, when run on a computer, execute the method as described in any one of claims 1 to 4.

Images