A method and apparatus for blow detection

Abstract

The embodiment of the present application provides a blowing detection method and device, relates to the technical field of electronics, can detect the multi-dimensional features such as the position, direction, intensity and rhythm of blowing of a user, performs various complex response operations or input controls according to the features, enhances the human-computer interaction capability, and improves the entertainment and user experience. The specific scheme is as follows: an electronic device collects a first image by using a first image collection device, collects a second image by using a second image collection device, and the second image is a thermal image; the blowing behavior of the user is identified according to the second image; the direction of blowing of the user is determined according to the first image and / or the second image; three-dimensional information of the face of the user relative to the electronic device is determined according to the first image and the second image; the target position of the blowing airflow reaching the electronic device is determined according to the blowing direction and the three-dimensional information; and a response is performed according to the features of the blowing behavior, and the features include the target position. The embodiment of the present application is used for blowing detection.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the electronic technical field, and in particular, to a blowing detection method and device. BACKGROUND

[0002] With the development of electronic technology, the interaction between users and electronic devices such as mobile phones is becoming more and more diversified. For example, users can use blowing to play some interesting games on the mobile phone. The mobile phone can detect the blowing of the user through the microphone.

[0003] In the existing blowing detection scheme, the microphone can periodically (for example, 10 ms) collect audio pulse data. The mobile phone modulates the audio pulse data, then squares the amplitude of the modulated waveform signal, and distinguishes whether the user is blowing according to the value of the square sum. For example, as shown in (a) of FIG. 1, the amplitude of the waveform signal detected when the user is speaking is large, but due to the discontinuity of the speaking process, the waveform distribution is sparse, and the square sum of the amplitude is small. As shown in (b) of FIG. 1, the amplitude of the waveform signal detected when blowing is relatively stable and continuous, the waveform is dense, similar to uniform distribution, and the square sum of the amplitude is large. Figure 1 Figure 1

[0004] The existing microphone detection scheme can detect whether the user is blowing, and the mobile phone can control the input according to whether the user is blowing, thereby achieving simple operation functions. SUMMARY

[0005] Embodiments of the present application provide a blowing detection method and device, which can detect the multi-dimensional features such as the position, direction, intensity, rhythm, etc. of the user's blowing, so as to perform various complex response operations or input controls according to the multi-dimensional features, enhance the human-computer interaction ability, and improve the entertainment and user experience.

[0006] To achieve the above object, the embodiments of the present application adopt the following technical solutions:

[0007] ​​In an aspect, an embodiment of the present application provides a method for detecting blowing, applied to an electronic device, the electronic device comprising a first image acquisition device and a second image acquisition device. The method comprises: the electronic device acquiring a first image and a second image. The first image is an image of a user's face captured by the first image acquisition device, and the second image is a thermal image of the user's face captured by the second image acquisition device. The electronic device identifies a blowing behavior of the user according to the second image. Then, the electronic device identifies an orientation of the user's face according to the first image and / or the second image. The electronic device determines a blowing direction of the user according to the orientation of the user's face. The electronic device determines three-dimensional information of the user's face relative to the electronic device according to the first image and the second image. The electronic device determines a target position of a blowing airflow reaching the electronic device according to the blowing direction and the three-dimensional information. The electronic device performs a response operation according to a feature of the blowing behavior, and the feature comprises the target position.

[0008] In this scheme, the electronic device can determine the three-dimensional information between the user's face and the electronic device when blowing by using the commonly used first image acquisition device in combination with the infrared camera (i.e. the second image acquisition device), and then determine the position of the airflow reaching the screen and the direction of the airflow according to the three-dimensional information, without the need for a special 3D camera, so that the blowing behavior of the user can be recognized by using fewer cameras.

[0009] In a possible design, the three-dimensional information of the user's face relative to the electronic device comprises three-dimensional information of the user's mouth relative to the first image acquisition device of the electronic device.

[0010] That is, the electronic device can determine the three-dimensional coordinate position of the user's mouth relative to the electronic device according to the first image and the second image.

[0011] In another possible design, the electronic device comprises a screen, and the electronic device identifies the blowing behavior of the user according to the second image, comprising: the electronic device identifies the blowing behavior of the user against the screen according to the second image.

[0012] That is, the electronic device can identify the blowing behavior of the user against a specific part of the electronic device, such as the screen, through the thermal image.

[0013] In another possible design, the method further comprises: the electronic device determining the direction of the airflow on the electronic device according to a plurality of target positions.

[0014] It can be understood that the direction indicated by the track formed by the plurality of positions of the electronic device reached by the airflow when blowing can be the direction of the airflow.

[0015] In another possible design, the electronic device identifies the blowing behavior of the user according to the second images, including: the electronic device identifies airflow images in the plurality of second images. The electronic device identifies the blowing behavior of the user according to area change rates of the airflow images between adjacent second images in the plurality of second images. The area change rate dS is: dS=(S2-S1) / T. S1 represents the area of the airflow image in a previous second image in the two adjacent collected second images; S2 represents the area of the airflow image in a subsequent second image; and T represents the collection interval of the two adjacent second images.

[0016] That is, the electronic device can identify the blowing behavior of the user according to the area change rates of the airflow images between the plurality of thermal images.

[0017] In another possible design, the electronic device identifies the blowing behavior of the user according to the area change rates of the airflow images between adjacent second images in the plurality of second images, including: if the average value of the area change rates of the airflow images between adjacent second images in the plurality of second images collected in the first detection period is greater than or equal to a first preset value, the electronic device determines that the blowing operation of the user is detected.

[0018] In this way, if the average value of the area change rates of the airflow images in a certain detection period is large, it can be indicated that the overall trend of the area change rates of the airflow images in this time period is large, and the overall trend of the change of the airflow state in this time period is also large, so the electronic device can determine that the user is blowing.

[0019] In another possible design, if the average value of the area change rates of the airflow images in the first detection period is greater than or equal to a second preset value, the blowing strength of the blowing operation is large. If the average value of the area change rates of the airflow images in the first detection period is greater than or equal to the first preset value and less than the second preset value, the blowing strength of the blowing operation is small.

[0020] It can be understood that when the average value of the area change rates is not only greater than the first preset value but also greater than the second preset value, it can be indicated that the area change rate is large and the airflow state changes rapidly, and the user may be blowing hard. When the average value is greater than the first preset value and less than or equal to the second preset value, it can be indicated that the area change rate is large and the airflow state changes rapidly, and the user may be blowing gently.

[0021] In another possible design, the electronic device identifies the blowing behavior of the user according to the area change rate of the airflow image between adjacent second images in the plurality of second images, and further includes: after the electronic device determines that the blowing operation of the user is detected, if the average value of the area change rate of the airflow image between adjacent second images in the plurality of second images collected in the second detection period is less than a third preset value, and the third preset value is less than or equal to the first preset value, the electronic device determines that the user stops blowing. The time length from when the electronic device determines that the blowing operation of the user is detected to when the electronic device determines that the user stops blowing is the blowing time length of the blowing operation.

[0022] It can be understood that when the average value is less than the third preset value, it can be indicated that the area of the airflow image changes less in the detection period, the airflow state changes slower in the detection period, and the user can have stopped blowing.

[0023] In another possible design, the electronic device identifies the blowing behavior of the user according to the area change rate of the airflow image between adjacent second images in the plurality of second images, and includes: if the electronic device detects that the area change rate of the airflow image between two second images collected adjacently is greater than or equal to the first preset value, the electronic device determines that the blowing operation of the user is detected. If the electronic device detects that the area change rate of the airflow image between two second images collected adjacently is less than a third preset value, and the third preset value is less than or equal to the first preset value, the electronic device determines that the user stops blowing.

[0024] That is, if the electronic device determines that the area change rate of the airflow image between two second images is greater than or equal to the first preset value, it can be indicated that the area of the airflow image changes greatly, the airflow state changes greatly, and the user is performing the blowing operation. If the area change rate of the airflow image between two second images is less than the third preset value, it can be indicated that the area of the airflow image changes less, the airflow state changes less, and the user can have stopped blowing.

[0025] In another possible design, the electronic device identifies the blowing behavior of the user according to the area change rate of the airflow image between adjacent second images in the plurality of second images, and further includes: the electronic device determines the blowing and stopping blowing conditions according to the change of the area change rate of the airflow image between adjacent second images in the plurality of second images. The electronic device determines the blowing mode of the user according to the blowing and stopping blowing conditions, and the blowing mode includes single blowing or continuous blowing for a plurality of times within a preset time length.

[0026] That is, the electronic device can determine the blowing mode of the user according to the area change rate to determine the related conditions when the user starts blowing and stops blowing.

[0027] In another possible design, the feature of the blowing behavior further includes one or more of a blowing direction, a blowing distance, an airflow direction, a blowing strength, a blowing duration, a blowing frequency, a blowing rhythm, or a blowing mode.

[0028] That is, the electronic device can also identify the multi-dimensional feature of the blowing behavior of the user according to the thermal image collected by the second image collection device, so as to perform different response operations or input controls according to the multi-dimensional feature, enhance the human-computer interaction capability, and improve the interest and user experience.

[0029] In another possible design, the electronic device performs a response operation according to the feature of the blowing behavior, including: the electronic device performs a response operation on a target object displayed at a target position according to the feature of the blowing behavior. Wherein, the feature is different, and one or more of a size, a color, a shape, a transparency, a type, a material, a number, a moving direction, a moving distance, a rotation angle, an action range, or a response number of the target object are also different.

[0030] That is, the electronic device can perform different response operations on the target object displayed at the target position according to different features of the blowing behavior.

[0031] In another possible design, the electronic device performs a response operation according to the feature of the blowing behavior, including: the electronic device performs a corresponding touch operation according to the feature of the blowing behavior. Wherein, the feature is different, and the type and / or operation mode of the touch operation are different; the type includes a click operation, a sliding operation, or a drag operation. The operation mode of the click operation includes one or more of a click position, a click number, a click strength, or a pressing duration. The operation mode of the sliding operation includes one or more of a sliding position, a sliding direction, or a sliding distance. The operation mode of the drag operation includes one or more of a dragged object, a drag direction, or a drag distance.

[0032] In this way, the electronic device can perform different touch operations according to different features of the blowing behavior.

[0033] In another possible design, the electronic device performs a response operation according to the feature of the blowing behavior, including: the electronic device scrolls a page according to the feature of the blowing behavior; and the feature is different, and the scrolling direction and / or scrolling amplitude of the page are different. Alternatively, the electronic device turns a page according to the feature of the blowing behavior; and the feature is different, and the order of turning the page forward or backward and / or the number of turning the page are different. The electronic device updates the interface content according to the feature of the blowing behavior; and the feature is different, and the update effect of the interface is different.

[0034] In this way, the electronic device can perform different page scrolling operations, page turning operations, or interface updating operations, etc. according to different features of the blowing behavior.

[0035] In another possible design, the electronic device responds to the characteristics of the blowing behavior, including: the electronic device responds with sound, and the characteristics are different, one or more of a tune, a pitch, a timbre, a scale, or a channel of the sound are also different. Or, the electronic device responds with light, and the characteristics are different, one or more of a color, a duration of lighting, a pattern projected, a number of light beams, or a frequency of flashing of the light are also different. Or, the electronic device responds with vibration, and the characteristics are different, one or more of a frequency, an amplitude, or a duration of the vibration are also different.

[0036] In this way, the electronic device can respond with different response manners of sound, light, or vibration according to different characteristics of the blowing behavior.

[0037] In another possible design, the electronic device responds to the characteristics of the blowing behavior, including: the electronic device responds with a variable mechanism, and the characteristics are different, one or more of a direction of extension and contraction, a degree of extension and contraction, a direction of rotation, a degree of rotation, a shape, a size, a material, or a color of the variable mechanism are also different.

[0038] In this way, the electronic device can respond with different variable hardware mechanisms provided on the electronic device according to different characteristics of the blowing behavior.

[0039] On the other hand, an embodiment of the present application provides a detection device, which is included in an electronic device, and has a function of implementing the behavior of the electronic device in any method in the above aspects and possible designs. The function can be implemented by hardware, or by executing corresponding software by hardware. The hardware or software includes at least one module or unit corresponding to the above function. For example, a collection module or unit, an identification module or unit, a determination module or unit, an execution module or unit, and a display module or unit, etc.

[0040] On the other hand, an embodiment of the present application provides an electronic device, including: one or more processors, a memory, a first image collection device, and a second image collection device. The first image collection device is configured to collect an image of a user's face; the second image collection device is configured to collect a thermal image of the user's face. The memory stores code; when the code is executed by the one or more processors, the electronic device performs the blowing detection method in any possible design of the above aspects.

[0041] On the other hand, an embodiment of the present application provides a computer storage medium, including computer instructions, when the computer instructions run on an electronic device, the electronic device performs the blowing detection method in any possible design of the above aspects.

[0042] In yet another aspect, the embodiments of the present application provide a computer program product, which, when running on a computer, causes the computer to perform the blow detection method in any possible design of the aspects above.

[0043] In yet another aspect, the embodiments of the present application provide a chip system, which comprises a processor and a memory; the memory stores a code; when the code is executed by the processor, the chip system performs the blow detection method in any possible design of the aspects above. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A blow detection method provided by the prior art is shown in a schematic diagram;

[0045] Figure 2 A hardware structure of a mobile phone provided by the embodiments of the present application is shown in a schematic diagram;

[0046] Figure 3 A software structure of a mobile phone provided by the embodiments of the present application is shown in a schematic diagram;

[0047] Figures 4A-4C A group of interface diagrams provided by the embodiments of the present application is shown in a schematic diagram;

[0048] Figures 5A-5D A group of area change rate diagrams of airflow images provided by the embodiments of the present application is shown in a schematic diagram;

[0049] Figures 6A-6B Another group of area change rate diagrams of airflow images provided by the embodiments of the present application is shown in a schematic diagram;

[0050] Figures 7A-7C A group of principle diagrams for determining a target position provided by the embodiments of the present application is shown in a schematic diagram;

[0051] Figure 8 A target position provided by the embodiments of the present application is shown in a schematic diagram;

[0052] Figures 9A-9D A group of airflow direction diagrams provided by the embodiments of the present application is shown in a schematic diagram;

[0053] Figures 10A-10C A group of blow response diagrams provided by the embodiments of the present application is shown in a schematic diagram;

[0054] Figures 11A-11C Another group of blow response diagrams provided by the embodiments of the present application is shown in a schematic diagram;

[0055] Figures 12A-12B Another group of blow response diagrams provided by the embodiments of the present application is shown in a schematic diagram;

[0056] Figure 13 A method flowchart of blow detection provided by the embodiments of the present application is shown in a schematic diagram;

[0057] Figure 14 Another method flowchart of the blowing detection provided by the embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, “ / ” represents the meaning of or, for example, A / B can represent A or B; “and / or” in this document is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” means two or more than two.

[0059] The embodiment of the present application provides a blowing detection method, which can be applied to an electronic device. The electronic device can include a first image acquisition device and a second image acquisition device. The first image acquisition device can be an optical camera device, and the second image acquisition device can be an infrared camera device. Alternatively, the first image acquisition device and the second image acquisition device can both be infrared camera devices. In the following embodiments of the present application, the first image acquisition device is taken as an optical camera device, and the second image acquisition device is taken as an infrared camera device as an example for description.

[0060] The optical camera device can include an image sensor, and one or more images can be captured by using visible light. The multiple images can also be video images. For example, the image sensor can be a charge coupled device (CCD), or a complementary metal-oxide-semiconductor (CMOS) phototransistor, etc. The optical camera device can be a color camera, a black-and-white camera, or a depth camera, etc.

[0061] The infrared camera device can include an infrared temperature sensor, and one or more heat distribution images, also known as thermal images, can be captured based on the principle of thermal imaging, by using the infrared energy radiation and temperature distribution of an object. For example, the infrared camera device can be an infrared thermal imager or other infrared sensing device.

[0062] When the user blows air towards the electronic device, the electronic device can detect the blowing behavior of the user according to a first image collected by a first image collection device and / or a second image collected by a second image collection device. The electronic device can determine three-dimensional information of the user's face relative to the electronic device based on the binocular ranging principle according to the first image and the second image. Further, the electronic device can determine the position of the airflow on the screen of the electronic device when blowing according to the three-dimensional information. In addition, the electronic device can also determine the blowing direction (or blowing angle) of the user according to the first image and / or the second image, determine the blowing strength, blowing duration, or blowing rhythm and other multi-dimensional features according to the second image. Thus, the electronic device can perform complex input control on the electronic device according to the multi-dimensional features of the blowing, enhance the human-computer interaction capability, and improve the entertainment and user experience.

[0063] In some embodiments, the user can blow air towards the electronic device, or blow air towards a specific part of the electronic device. For example, the specific part can be the screen, the back cover, the air blowing sensitive part, or other parts of the electronic device. When the user blows air towards the screen of the electronic device, the electronic device can display different user interfaces on the screen in response to the blowing operation of the user. In the following embodiments of the present application, the electronic device detecting the user blowing air towards the screen will be taken as an example for illustration.

[0064] For example, the electronic device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like. The device type of the electronic device is not specifically limited in the embodiments of the present application.

[0065] For example, when the electronic device is a mobile phone, Figure 2A structural schematic diagram of the mobile phone 100 is shown. The mobile phone 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an infrared thermal imager 196, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, or an ultrasonic sensor, etc.

[0066] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0067] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated into one or more processors.

[0068] The controller can be the nerve center and command center of the mobile phone 100. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of instruction fetching and instruction execution.

[0069] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is cycling through. If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. This avoids repeated access and reduces the latency of the processor 110, thus improving the efficiency of the system.

[0070] The camera 193 can be the first image acquisition device described above, and the infrared thermal imager 196 can be the second image acquisition device described above.

[0071] In some embodiments, the processor 110 can identify the image collected by the camera 193 to determine the orientation of the user's face, so as to determine the blowing direction of the user according to the orientation of the user's face.

[0072] In some embodiments, the processor 110 can also determine the position of the user's mouth relative to the first image acquisition device according to the image collected by the camera 193 and the image collected by the infrared thermal imager 196.

[0073] In some embodiments, the processor 110 can include one or more interfaces. The interface can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0074] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C bus. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, and the touch function of the mobile phone 100 is realized.

[0075] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can contain multiple sets of I2S bus. The processor 110 can be coupled to the audio module 170 through the I2S bus, and communication between the processor 110 and the audio module 170 is realized. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, and the function of answering the phone through the Bluetooth earphone is realized.

[0076] The PCM interface can also be used for audio communication, which samples, quantizes and encodes analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, and the function of answering the phone through the Bluetooth earphone is realized. Both the I2S interface and the PCM interface can be used for audio communication.

[0077] The UART interface is a universal serial data bus, which is used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, and the Bluetooth function is realized. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, and the function of playing music through the Bluetooth earphone is realized.

[0078] The MIPI interface can be used to connect the processor 110 and the display screen 194, the camera 193 and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to realize the shooting function of the mobile phone 100. The processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the mobile phone 100.

[0079] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0080] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 can be used to connect a charger to charge the mobile phone 100, or to transmit data between the mobile phone 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other electronic devices, such as AR devices and the like.

[0081] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the mobile phone 100. In other embodiments of the present application, the mobile phone 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0082] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the mobile phone 100. The charging management module 140 can charge the battery 142 while also providing power to electronic devices through the power management module 141.

[0083] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as the battery capacity, the number of battery cycles, the battery health status (leakage, impedance), and the like. In some embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can be disposed in the same device.

[0084] The wireless communication function of the mobile phone 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0085] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0086] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the mobile phone 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and the like the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor and convert the signals into electromagnetic waves radiated by the antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0087] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a microphone 170B, etc.), or displays an image or a video through the display screen 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0088] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, modulate it, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.

[0089] In some embodiments, the antennas 1 and the mobile communication module 150 of the mobile phone 100 are coupled, and the antennas 2 and the wireless communication module 160 are coupled, so that the mobile phone 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0090] The mobile phone 100 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0091] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the mobile phone 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0092] The mobile phone 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0093] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also optimize the algorithm for the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0094] The camera 193 is configured to capture still images or videos. For example, it can be a color camera, a black-and-white camera, an infrared camera, or a depth camera, and the like. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the mobile phone 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0095] The camera 193 can be arranged at the top, bottom, side frame, or inside the display 194 of the mobile phone 100, and can be a fixed camera or a liftable camera. The arrangement position and arrangement manner of the camera 193 are not limited in the embodiments of the present application.

[0096] In some embodiments, the camera 193, the infrared thermal imager 196, and the display are located in the same plane.

[0097] In some embodiments, when the user blows, the camera 193 can capture the face image of the user to identify the direction of the face of the user in the image, so as to determine the blowing direction of the user.

[0098] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile phone 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0099] The video codec is used to compress or decompress digital video. The mobile phone 100 can support one or more video codecs. In this way, the mobile phone 100 can play or record videos in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0100] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by referring to the structure of a biological neural network, such as the transmission mode between human brain neurons, and can also continuously self-learn. Through the NPU, the mobile phone 100 can realize intelligent cognition and other applications, such as image recognition, face recognition, voice recognition, text understanding, and the like.

[0101] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, and other files can be saved in the external memory card.

[0102] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 performs various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data (such as audio data, a phone book, etc.) created during use of the mobile phone 100, and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0103] The mobile phone 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, the application processor, and the like. For example, music playing, recording, and the like.

[0104] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or part of the functions of the audio module 170 can be disposed in the processor 110.

[0105] The speaker 170A, also known as a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The mobile phone 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0106] The receiver 170B, also known as a “earpiece”, is used to convert an audio electrical signal into a sound signal. When the mobile phone 100 answers a call or a voice message, the receiver 170B can be held close to the ear of a person to listen to the voice.

[0107] The microphone 170C, also known as a “microphone”, “sound transducer”, is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, a user can speak into the microphone 170C close to the mouth to input a sound signal into the microphone 170C. The mobile phone 100 can be provided with at least one microphone 170C. In other embodiments, the mobile phone 100 can be provided with two microphones 170C, which can not only collect sound signals, but also realize a noise reduction function. In other embodiments, the mobile phone 100 can be provided with three, four or more microphones 170C, which can not only collect sound signals and reduce noise, but also identify the source of the sound, realize directional recording functions, and the like.

[0108] The earphone interface 170D is used to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0109] The pressure sensor 180A is used to sense a pressure signal, and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. A capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The phone 100 determines the intensity of the force according to the change in capacitance. When a touch operation is applied to the display screen 194, the phone 100 detects the intensity of the touch operation according to the pressure sensor 180A. The phone 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed. In some embodiments, the pressure sensor can detect the change in pressure of a local area of the screen when the user blows air.

[0110] The gyroscope sensor 180B can be used to determine the motion attitude of the phone 100. In some embodiments, the angular velocity of the phone 100 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the phone 100, and calculates the distance that the lens module needs to compensate according to the angle, so that the lens counteracts the shaking of the phone 100 by moving in the opposite direction, thereby achieving anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0111] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the phone 100 calculates the altitude, assists in positioning and navigation by measuring the air pressure value with the barometric pressure sensor 180C.

[0112] The magnetic sensor 180D includes a Hall sensor. The mobile phone 100 can detect the opening and closing of a flip cover with the magnetic sensor 180D. In some embodiments, when the mobile phone 100 is a flip phone, the mobile phone 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. In turn, according to the detected opening and closing state of the cover or the opening and closing state of the flip cover, the mobile phone 100 can set a flip cover automatic unlocking feature or the like.

[0113] The acceleration sensor 180E can detect the magnitude of acceleration of the mobile phone 100 in various directions (typically three axes). When the mobile phone 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device, applied to landscape / portrait screen switching, pedometer applications, and the like.

[0114] The distance sensor 180F is used to measure distance. The mobile phone 100 can measure distance by infrared or laser. In some embodiments, the mobile phone 100 can use the distance sensor 180F to measure distance to achieve fast focusing when shooting a scene.

[0115] The proximity light sensor 180G can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The mobile phone 100 emits infrared light outwardly through the light-emitting diode. The mobile phone 100 detects infrared reflected light from nearby objects using the photodiode. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone 100. When insufficient reflected light is detected, the mobile phone 100 can determine that there is no object near the mobile phone 100. The mobile phone 100 can use the proximity light sensor 180G to detect that a user is holding the mobile phone 100 close to the ear for a call, so as to automatically turn off the screen to achieve power saving. The proximity light sensor 180G can also be used for automatic unlocking and locking of the cover in cover mode and pocket mode.

[0116] The ambient light sensor 180L is used to sense ambient light brightness. The mobile phone 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking a picture. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the mobile phone 100 is in a pocket to prevent accidental touch.

[0117] The fingerprint sensor 180H is used to collect fingerprints. The mobile phone 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locking, fingerprint shooting, fingerprint answering a call, and the like.

[0118] The temperature sensor 180J is configured to detect temperature. In some embodiments, the phone 100 utilizes the temperature detected by the temperature sensor 180J to implement temperature handling strategies. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the phone 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In another example, when the temperature is below another threshold, the phone 100 heats the battery 142 to avoid abnormal shutdown of the phone 100 caused by low temperature. In yet another example, when the temperature is below yet another threshold, the phone 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature. In some embodiments, the temperature sensor can be a thermistor, which can detect the temperature change of a local area of the screen caused by air flow when a user blows air.

[0119] The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch panel", or simply a screen. The touch sensor 180K is configured to detect a touch operation acting on or near the touch sensor 180K. The touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In another example, the touch sensor 180K can also be disposed on the surface of the phone 100, which is different from the position of the display screen 194.

[0120] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a human body sound part vibration bone block. The bone conduction sensor 180M can also contact the human body pulse to receive a blood pressure pulsation signal. In some embodiments, the bone conduction sensor 180M can also be disposed in a headset to form a bone conduction headset. The audio module 170 can analyze a voice signal based on the vibration signal of the sound part vibration bone block obtained by the bone conduction sensor 180M to implement a voice function. The application processor can analyze heart rate information based on the blood pressure pulsation signal obtained by the bone conduction sensor 180M to implement a heart rate detection function.

[0121] The keys 190 include a power on key, a volume key, and the like. The keys 190 can be mechanical keys. They can also be touch keys. The phone 100 can receive key inputs to generate key signal inputs related to user settings and function control of the phone 100.

[0122] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations acting on different regions of the display screen 194 can also correspond to different vibration feedback effects of the motor 191. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0123] The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0124] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the mobile phone 100. The mobile phone 100 can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. The same SIM card interface 195 can simultaneously insert multiple cards. The types of multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The mobile phone 100 interacts with the network through the SIM card to realize functions such as calling and data communication. In some embodiments, the mobile phone 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone 100 and cannot be separated from the mobile phone 100.

[0125] The infrared thermal imager 196 can use an infrared detector and an optical imaging objective lens to receive infrared radiation of a measured target. The energy distribution pattern of the measured target is reflected on the photosensitive element of the infrared detector, thereby forming an infrared thermal image. The thermal image can record the heat or temperature radiated by the object itself. The thermal image corresponds to the thermal distribution field of the object surface. That is, the infrared thermal imager 196 can use infrared energy and temperature distribution to shoot a heat distribution image, thereby converting invisible infrared energy emitted by the object into a visible thermal image.

[0126] The object surface continuously radiates infrared rays. Infrared rays can reflect the infrared radiation field of the object surface, i.e., the temperature field. Different objects have different infrared radiation fields of the object surface. The mobile phone 100 can identify different shooting objects according to different infrared radiation fields presented on the image collected by the infrared thermal imager 196. For example, the mobile phone 100 can identify the airflow blown by the user.

[0127] And, the object temperature is different, its emitted infrared energy is also different. The higher the temperature, the greater the emitted infrared energy. Different infrared energy is different in color on the image collected by the infrared thermal imager 196, that is, different colors on the thermal image represent different temperatures of the measured object. The mobile phone 100 can determine the temperature of different objects according to different colors on the image collected by the infrared thermal imager 196.

[0128] The infrared thermal imager 196 can be arranged at the top, bottom, side frame, or inside the display screen 194 of the mobile phone 100, and can be one or more. The type and position of the infrared thermal imager 196 are not limited in the embodiments of the present application.

[0129] The software system of the mobile phone 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the mobile phone 100.

[0130] Figure 3 is a software structure block diagram of the mobile phone 100 in the embodiments of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and the system library, and the kernel layer. The application layer can include a series of application packages.

[0131] As shown in Figure 3 , the application package can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like.

[0132] The application framework layer provides the application layer with an application programming interface (API) and a programming framework for the application. The application framework layer includes some pre-defined functions.

[0133] As shown in Figure 3 , the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0134] The window manager is used to manage the window program. The window manager can obtain the display screen size, judge whether there is a status bar, lock the screen, and intercept the screen, etc.

[0135] The content provider stores and retrieves data and makes it accessible to applications. The data can include videos, images, audio, dialed and received calls, browsing history and bookmarks, phonebook, etc.

[0136] The view system includes visual controls, such as controls that display text, controls that display pictures, etc. The view system can be used to build an application. A display interface can be composed of one or more views. For example, a display interface that includes a short message notification icon can include a view that displays text and a view that displays a picture.

[0137] The telephony manager is used to provide the communication functions of the phone 100. For example, management of call status (including call connection, call hang-up, etc.).

[0138] The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0139] The notification manager enables applications to display notification information in the status bar. The notification manager can be used to convey messages of the notification type, which can automatically disappear after a short period of time without user interaction. For example, the notification manager is used to notify of download completion, message reminders, etc. The notification manager can also be a notification that appears in the top status bar of the system in the form of a graph or a scrolling text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, the phone 100 vibrates, the indicator light blinks, etc.

[0140] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0141] The core library includes two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.

[0142] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0143] The system library can include multiple functional modules. For example: a surface manager, media libraries, a three-dimensional graphics processing library (for example: OpenGL ES), a 2D graphics engine (for example: SGL), etc.

[0144] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0145] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0146] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0147] A 2D graphics engine is a graphics engine for 2D drawing.

[0148] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0149] In the embodiments of this application, after the camera 193 and the infrared thermal imager 196 acquire images, the hardware driver in the kernel layer can obtain the images and report them to the processor 110. The processor 110 processes the images acquired by the camera 193 and the infrared thermal imager 196 to detect the characteristics of the blowing behavior, including the position of the airflow on the display screen 194, the airflow direction, the blowing direction, the blowing duration, the blowing intensity, or the blowing mode, and other multi-dimensional features. The processor 110 reports the multi-dimensional features of the blowing to the application framework layer. The application framework layer can notify relevant applications (e.g., applications that support blowing operations) through Android message broadcasting or event notification. The relevant applications execute corresponding response operations according to the preset correspondence between the response operations and the multi-dimensional features of the blowing, thereby realizing multi-dimensional input control of the blowing.

[0150] The following will use electronic devices as an example. Figure 2 and Figure 3 The mobile phone 100 with the structure shown has a first image acquisition device of camera 193 and a second image acquisition device of infrared thermal imager 196. The mobile phone 100 detects the user blowing air towards the screen of the mobile phone 100. Taking the screen as a display screen 194 or a touch screen as an example, the blowing detection method provided in the embodiment of this application will be described.

[0151] The mobile phone can activate the camera and infrared thermal imager, and use the camera and infrared thermal imager to capture images in real time.

[0152] For example, the mobile phone can periodically collect a plurality of sets of target images by using the camera and the infrared thermal imager. The plurality of sets of target images include a plurality of first images and a plurality of second images. Each set of target images can be collected in each collection period T. Each set of target images includes a first image collected by the camera for a face of the user and a second image collected by the infrared thermal imager for the face of the user. The second image is a thermal image. The first image and the second image in the same set of target images can be images collected by the mobile phone at the same time. For example, the collection period T is 15 ms.

[0153] The mobile phone can start the camera and the infrared thermal imager in various ways. For example, in some embodiments, the mobile phone can start the camera and the infrared thermal imager after being powered on and periodically collect images so that the blowing operation of the user can be detected in time according to the collected images.

[0154] In other embodiments, the mobile phone can start the camera and the infrared thermal imager and periodically collect images after being unlocked because the user generally performs the blowing operation after the mobile phone is unlocked. When not unlocked, the mobile phone can not start or stop the camera and the infrared thermal imager to reduce the power consumption of the mobile phone.

[0155] In yet other embodiments, the mobile phone can start the camera and the infrared thermal imager and periodically collect images after an application supporting the blowing operation is opened. The mobile phone can stop the camera and the infrared thermal imager after the application supporting the blowing operation is closed to reduce the power consumption of the mobile phone.

[0156] For example, the wind chime application supports the blowing operation. The mobile phone can open the wind chime application and automatically start the camera and the infrared thermal imager after detecting that the user clicks the icon 401 as shown in FIG. 4A. Figure 4A

[0157] In other embodiments, the mobile phone can start the camera after being powered on and periodically collect images by using the camera. When the camera collects an image of the face of the user or the mouth of the user, the user can perform the blowing operation, and thus the infrared thermal imager can be started and images can be periodically collected by using the camera and the infrared thermal imager.

[0158] In other embodiments, the mobile phone can start the infrared thermal imager after being powered on and periodically collect thermal images by using the infrared thermal imager. The mobile phone can start the camera after determining that the blowing operation of the user is detected according to the thermal images, and thus determine the characteristics of the blowing operation according to the images collected by the camera and the infrared thermal imager.

[0159] In yet other embodiments, the mobile phone can start the camera and the infrared thermal imager and periodically collect images after detecting an operation of the user indicating that the blowing function is opened. In yet other embodiments, the mobile phone can start the camera and the infrared thermal imager and periodically collect images after detecting an operation of the user indicating that the blowing function is opened.

[0160] For example, as shown in FIG. 3A, the setting interface of the mobile phone has a function switch of the blow operation. After detecting an operation of the user indicating to turn on the switch, the mobile phone turns on the camera and the infrared thermal imager, and periodically collects images. Figure 4B

[0161] For another example, after detecting a preset operation of the user indicating to turn on the blow function, the mobile phone turns on the camera and the infrared thermal imager, and periodically collects images. For example, as shown in FIG. 3B, after detecting that the user draws a "cq" track on the screen in the black screen state, the mobile phone turns on the camera and the infrared thermal imager. Figure 4C

[0162] For another example, after detecting an operation of the user indicating to turn on the blow function, or an operation of the user indicating to turn on the camera and the infrared thermal imager, the mobile phone turns on the camera and the infrared thermal imager, and periodically collects images.

[0163] In some other embodiments, after detecting an operation of the user indicating to turn off the blow function, the mobile phone can turn off the camera and the infrared thermal imager, and stop collecting images by using the camera and the infrared thermal imager.

[0164] Specifically, in the same period, the mobile phone can simultaneously collect images by using the camera and the infrared thermal imager, and collect a face image at the same time.

[0165] The mobile phone can detect a blow behavior of the user and a multi-dimensional feature of the blow behavior according to the plurality of first images and the plurality of second images.

[0166] In some embodiments, the mobile phone can determine whether the user blows according to the first image collected by the camera.

[0167] When blowing, the user usually puffs up the face and purses the mouth, and the face and the mouth of the user change accordingly. Thus, the mobile phone can extract the facial feature and the mouth feature of the user on the first image by using an image recognition algorithm. When the facial feature and the mouth feature of the user on the first image match the facial feature and the mouth feature of the user when actually blowing, the mobile phone determines that the blow operation of the user is detected.

[0168] For example, the image recognition algorithm can include an artificial intelligence (AI) classification algorithm such as a neural network algorithm, a pattern recognition algorithm, and other recognition algorithms. The feature extraction algorithm can include a variety of methods such as a feature extraction method based on a static image (for example, a global method or a local method), or a feature extraction method based on a dynamic video image (for example, an optical flow method, a model method, or a geometric method). The type of image processing algorithm such as the image recognition algorithm and the feature extraction algorithm is not limited in the embodiments of the present application.

[0169] ​​The neural network algorithm can include a convolutional neural network (CNN) algorithm, a recurrent neural network (RNN) algorithm, a back propagation (BP) neural network algorithm, or a radial basis function (RBF) neural network algorithm, or the like. The neural network needs to be learned and trained according to certain learning criteria before it can work.

[0170] For example, after the neural network is trained and the first image is input into the neural network, if the label output by the neural network indicates "blowing", the mobile phone can determine that the facial features and the mouth features of the user in the first image match the facial features and the mouth features of the user when the user actually blows, and the user is blowing towards the screen.

[0171] For another example, the pattern recognition algorithm can process and analyze various forms of information (for example, images) representing things or phenomena, to describe and classify the things or phenomena, and the like. For example, a K-nearest neighbor algorithm, a bayes classifier algorithm, a principle component analysis (PCA) algorithm, a linear discriminant analysis (LDA) algorithm, a non-negative matrix factorization (NMF) algorithm, a gaussian mixture model (GMM), or the like can be used to recognize the shooting scene. When the mobile phone determines that the mouth features of the user in the first image match the preset mouth features through the pattern recognition algorithm, it can be determined that the mouth features of the user in the first image match the mouth features of the user when the user actually blows, and the user is blowing towards the screen.

[0172] In addition, when the user blows, other parts of the face, in addition to the face and the mouth, will also change accordingly, for example, the chin, the nose, or the eyes, and the like. Therefore, the mobile phone can also determine whether the user is blowing by combining the features of other parts of the face, to improve the blowing detection accuracy. When the facial features and the mouth features of the user in the first image match the facial features and the mouth features of the user when the user actually blows, and the features of other parts of the face in the first image match the features of other parts of the face of the user when the user actually blows, the mobile phone determines that the blowing operation of the user is detected.

[0173] The mobile phone can also determine whether the user stops blowing and the blowing duration, blowing strength, or blowing rhythm according to the features of the face part in the first image. For example, the greater the degree of the user puffing his face and pursing his lips in the first image, the greater the blowing strength. For another example, the smaller the degree of the user puffing his face and pursing his lips in the first image, the smaller the blowing strength. For another example, the mobile phone determines that the user blows according to the features of the user puffing his face and pursing his lips in the first image, and the mobile phone determines that the user stops blowing according to the features of the user stopping puffing his face and pursing his lips in the first image. The duration between the user starting to puff his face and purse his lips and the user stopping puffing his face and purse his lips is the blowing duration between the user starting to blow and stopping blowing. For another example, the mobile phone can determine that the user blows twice in succession according to the features of the user puffing his face and pursing his lips twice in succession in the first image.

[0174] In some other embodiments, the mobile phone can determine whether the user blows according to the second image collected by the infrared thermal imager.

[0175] The airflow exhaled by the user's mouth is usually emitted at a position outward from the user's mouth, and the temperature distribution of the airflow exhaled by the user's mouth conforms to a certain rule. According to the rule, the mobile phone can identify the airflow image in the second image, i.e., the image of the airflow exhaled by the user's mouth formed in the second image.

[0176] Compared with normal breathing, the initial speed, temperature, and gas volume of the airflow exhaled by the user when blowing are different. Thus, the airflow image formed in the second image when the user blows is different from the airflow image formed in the second image when the user breathes normally.

[0177] When the user breathes normally, the initial speed of the exhaled airflow is small, the distance moved by the airflow per unit time is short, and the state of the airflow changes slowly. Thus, the area change of the airflow image between the adjacent second images collected by the mobile phone (i.e., the second images collected in two adjacent collection periods T) is also small, i.e., the area change rate of the airflow image in the adjacent second images is small.

[0178] The area change rate dS can be expressed as: dS = (S2-S1) / T. S1 represents the area of the airflow image in the first second image of the two second images collected in succession, S2 represents the area of the airflow image in the second second image, and T represents the collection interval of the two second images, i.e., the collection period T.

[0179] Compared with normal breathing, the initial speed of the airflow exhaled by the user when blowing is large, the distance moved by the airflow per unit time is far, and the state of the airflow changes quickly. Thus, the area change of the airflow image between the second images collected by the mobile phone (or between the second images collected in succession by the mobile phone) is also large, i.e., the area change rate of the airflow image in the adjacent second images is large.

[0180] Therefore, the mobile phone can calculate the area change rate of the airflow image between the current collected second image and the last collected second image in real time. Thus, the blowing behavior of the user can be identified according to the area change rate of the airflow image.

[0181] In some embodiments, the area change rate of the airflow image can be represented by the change rate of the length and / or width of the airflow image. If the change rate of the length and / or width of the airflow image is large, it can be indicated that the area change rate of the airflow image is large. If the change rate of the length and / or width of the airflow image is small, it can be indicated that the area change rate of the airflow image is small.

[0182] In some embodiments, the area change rate of the airflow image can be represented by the change rate of the length and / or width of the airflow image. If the change rate of the length and / or width of the airflow image is large, it can be indicated that the area change rate of the airflow image is large. If the change rate of the length and / or width of the airflow image is small, it can be indicated that the area change rate of the airflow image is small.

[0183] In some embodiments, if the mobile phone detects that the area change rate of the airflow image between the two adjacent collected second images is greater than or equal to the first preset value, the mobile phone determines that the blowing operation of the user is detected.

[0184] That is, if the mobile phone determines that the area change rate of the airflow image between two second images is greater than or equal to the first preset value, it can be indicated that the area change of the airflow image is large, the airflow state change is large, and the user is blowing. In this scheme, the mobile phone can quickly and timely determine whether the user is blowing.

[0185] In some embodiments, if the mobile phone detects that the area change rate of the airflow image between the two adjacent collected second images is greater than or equal to the first preset value, the mobile phone determines that the blowing operation of the user is detected.

[0186] The detection period is greater than the acquisition period T of the image. For example, the acquisition period can be 15 ms, and the detection period can be 100 ms. The mobile phone can calculate the average value of the area change rate of the airflow image between the second images in the detection period (i.e., in the 100 ms before the current time) in real time. When the average value of the area change rate of the airflow image in a certain detection period is greater than or equal to the first preset value, it can be indicated that the overall trend of the area change rate of the airflow image in this time period is large, and the overall trend of the change of the airflow state in this time period is also large, so the mobile phone can determine that the user is blowing. In this scheme, the mobile phone can more accurately determine whether the user is blowing according to the average value of the area change rate, and avoid misjudgment caused by determining whether to blow according to one area change rate.

[0187] For example, Figures 5A-6B A curve of an average value of an area change rate is shown. The curve represents the average value of the area change rate of the airflow image in the 100 ms before each time point shown in the time axis (i.e., in the detection period).

[0188] For example, referring to Figure 5A For example, the average value of the area change rate of the airflow image in the 100 ms before the 215th ms (i.e., between the 115th ms and the 215th ms) is greater than the first preset value, and the mobile phone determines that the user is blowing.

[0189] In addition, the mobile phone can also detect multi-dimensional features of the blowing behavior according to the area change rate of the airflow image between the second images. For example, the blowing strength, blowing duration, blowing mode, blowing frequency, or blowing rhythm of the blowing operation.

[0190] For example, the mobile phone can detect the blowing strength. In some embodiments, if the mobile phone determines that the average value of the area change rate of the airflow image in the first detection period is greater than or equal to a second preset value, and the second preset value is greater than the first preset value, it is determined that the blowing operation of the user is detected and the blowing strength is large, that is, the mobile phone detects the strong blowing operation of the user. If the mobile phone determines that the average value of the area change rate of the airflow image in the first detection period is greater than or equal to the first preset value and less than the second preset value, it is determined that the blowing operation of the user is detected and the blowing strength is small, that is, the mobile phone detects the light blowing operation of the user.

[0191] When the average of the area change rate of the airflow image in a certain detection period is greater than a first preset value, it can be determined that the user's blowing operation is detected. When the average is also greater than a second preset value, it can be indicated that the area change rate is large and the airflow state changes quickly, and the user can be blowing hard. When the average is greater than the first preset value and less than or equal to the second preset value, it can be indicated that the area change rate is relatively large and the airflow state changes relatively quickly, and the user can be blowing gently.

[0192] For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard. Figure 5B For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard.

[0193] Figure 5C For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard.

[0194] When the average of the area change rate of the airflow image in a certain detection period is less than the third preset value, it can be indicated that the area of the airflow image changes little in the detection period, and the airflow state changes slowly in the detection period, and the user can have stopped blowing.

[0195] For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard.

[0196] For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard. Figure 5D For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard.

[0197] For example, as shown in FIG. 2, in the first 100 ms (i.e., between the 115th ms and the 215th ms), the average of the area change rate of the airflow image is greater than the first preset value, and the phone determines that the user is blowing hard.

[0198] ​According to the above description, the mobile phone can calculate the area change rate of the airflow image on the two adjacent second images in real time, and can also calculate the average value of the area change rate of the airflow image in each detection period in real time. The mobile phone can also determine the number of times, frequency or rhythm of blowing of the user according to the change of the area change rate of the airflow image.

[0199] For example, the mobile phone can determine the blowing and stopping blowing of the mobile phone according to the change of the area change rate of the airflow image. The mobile phone determines the blowing mode of the user according to the blowing and stopping blowing of the mobile phone. The blowing mode includes single blowing or continuous blowing multiple times within a preset time length. For example, if the mobile phone detects the blowing operation of the user within a preset time length 2 according to the size change of the area change rate of the airflow image, detects that the user stops blowing, and detects the blowing operation of the user again, the mobile phone can determine that the user blows twice continuously.

[0200] The preset time length 2 is greater than the detection period T. For example, the preset time length 2 can be 1s. For example, referring to the curve shown in FIG. 2, if the mobile phone detects that the user stops blowing at t3 within 1s after detecting that the user blows at t2, and then does not detect that the user blows again within the 1s, the mobile phone can determine that the user blows once. Figure 6A

[0201] For example, referring to the curve shown in FIG. 2, if the mobile phone detects that the user stops blowing at t5 within 1s after detecting that the user blows at t4, and then detects that the user blows again at t6 within the 1s, the mobile phone can determine that the user blows twice continuously. Figure 6B Similarly, the mobile phone can also detect the continuous blowing, blowing times, blowing frequency or blowing rhythm of the user according to the blowing and stopping blowing of the mobile phone. For example, the mobile phone can detect that the user blows fast first and then blows slowly, or blows twice lightly and then blows once with great force, according to the change of the area change rate of the airflow image.

[0202] In other embodiments, the mobile phone can identify the blowing behavior of the user according to the pattern of the airflow image on the second image.

[0203]

[0204] ​​On the second image, the first pattern of the airflow image corresponding to the airflow when the user breathes normally is different from the second pattern of the airflow image when the user blows air. For example, when the camera and the infrared thermal imager are arranged on the top of the mobile phone (i.e. the forehead part of the mobile phone), the first pattern is similar to an ellipse when the user blows air against the screen of the mobile phone; the second pattern is similar to a cone. In addition, due to the randomness and irregularity of natural wind, the pattern of the airflow image on the second image formed by the natural wind is usually irregular. The mobile phone can determine whether the user blows air according to the pattern of the airflow image on the second image.

[0205] For example, if the similarity between the pattern of the airflow image on the second image and the preset pattern is greater than or equal to a preset threshold 1, the mobile phone can determine that the airflow image is formed when the user blows air, and the mobile phone detects the blowing operation of the user.

[0206] For another example, if the average value of the similarity between the pattern of the airflow image on the second image collected in a certain detection period and the preset pattern is greater than or equal to a preset threshold 1, the mobile phone can determine that the airflow image is formed when the user blows air, and the mobile phone detects the blowing operation of the user.

[0207] Then, if the average value of the similarity between the pattern of the airflow image on the second image collected in a certain detection period and the preset pattern is less than or equal to a preset threshold 2, the mobile phone can determine that the airflow image is not formed when the user blows air, and the mobile phone determines that the user stops blowing air. The preset threshold 2 is less than or equal to the preset threshold 1.

[0208] For example, the infrared thermal imager is arranged on the top of the mobile phone, and the preset pattern is similar to a cone. If the pattern of the airflow image on the adjacent second images changes from similar to an ellipse to similar to a cone, the mobile phone determines that the blowing of the user is detected. If the pattern of the airflow image on the adjacent second images changes from similar to a cone to similar to an ellipse, the mobile phone determines that the user stops blowing air. The mobile phone determines the length of time between the time when the user starts blowing air and the time when the user stops blowing air as the blowing length.

[0209] The mobile phone can also determine the blowing strength of the user according to the pattern of the airflow image on the second image. The specific shape of the airflow image is different when the blowing strength is different. When the infrared thermal imager is arranged on the top of the mobile phone and the user blows air towards the screen, the pattern of the airflow image is similar to a cone; the greater the blowing strength, the greater the height of the cone; the smaller the blowing strength, the smaller the height of the cone. Thus, if the similarity between the pattern of the airflow image on the second image and a preset pattern is greater than or equal to a preset threshold 1, the pattern of the airflow image is similar to a cone shape, and the height of the cone is greater than or equal to a preset height value 1, it can be determined that the blowing strength of the user is greater. If the similarity between the pattern of the airflow image on the second image and a preset pattern is greater than or equal to a preset threshold 1, the pattern of the airflow image is similar to a cone shape, and the height of the cone is less than a preset height value 1, it can be determined that the blowing strength of the user is smaller.

[0210] In addition, the mobile phone can also determine the start of blowing, the stop of blowing, the blowing duration, the stop of blowing interval and the like of the user according to the pattern change of the airflow image on the second image, so as to determine the blowing times and the blowing rhythm of the user.

[0211] In other embodiments, the mobile phone can combine any combination of the facial features, the mouth features and the features of other parts of the face on the first image, the area change rate of the airflow image on the second image, or the shape of the airflow image on the second image to detect the blowing operation and the blowing features of the user, so as to improve the accuracy of blowing detection.

[0212] After detecting the blowing operation of the user, the mobile phone can determine the position (i.e., the target position) of the airflow blown by the user on the screen in real time according to the first image and the second image in each group of target images. The following will be specifically described.

[0213] The mobile phone can determine the three-dimensional (3D) information of the face of the user relative to the mobile phone according to the first image and the second image in each group of target images based on the binocular distance measurement principle. That is, the three-dimensional information of the face of the user relative to the mobile phone. For example, the three-dimensional information of a specific part (e.g., the mouth) on the face of the user relative to the mobile phone; or the three-dimensional information between a specific part on the face of the user and a specific part on the mobile phone, and the like. The following will be described by taking the three-dimensional information of the mouth on the face of the user relative to the camera on the mobile phone as an example.

[0214] The three-dimensional information can include the position parameter of the mouth relative to the camera when the user blows air. The position parameter can include the coordinates of the mouth in a three-dimensional rectangular coordinate system with the camera as the coordinate origin, that is, the three-dimensional distance of the mouth relative to the camera.

[0215] For example, referring to Figure 7AThe camera is located at point L, and the infrared thermal imager is located at point R. A three-dimensional rectangular coordinate system is established with L as the origin.

[0216] like Figure 7A As shown, on the XZ plane, the mouth is located at point P, with coordinates (x, z). According to the principle of similar triangles, we can obtain Equation 1: f / z = xl / x = xr / (xb). Where f represents the focal length of the camera, and b represents the distance between L and R. Figure 7C As shown, xl represents the position of a preset face part (e.g., mouth) on the first image, and the distance between it and the midline passing through the center of the first image and perpendicular to LR. xr represents the position of the preset face part (e.g., mouth) on the second image, and the distance between it and the midline passing through the center of the first image and perpendicular to LR. d = xl – xr represents the parallax of the first and second images in the X direction.

[0217] From Equation 1, we can derive Equation 2: x = xl * b / d; and Equation 3: z = f * b / d. Therefore, we can obtain the coordinates x and z of point P. Here, z represents the distance between the user's mouth and the plane of the screen when blowing air, i.e., the depth of the mouth relative to the screen when blowing air.

[0218] The mobile phone can calculate θ using Equation 4: θ = arctan(x / z). θ represents the angle between PL and PQ when the user's face is directly facing the screen.

[0219] The mobile phone can also identify the orientation of the user's face based on the first image and / or the second image using an image recognition algorithm. The phone can then identify the angle at which the user's face turns when blowing air, and the direction defined by this angle is the direction of the user's blowing. This angle can include a horizontal angle θ' and a vertical angle φ'.

[0220] Then, the phone can calculate the coordinates (x, y) of the intersection point F of the extended line of the blowing direction and the plane where the screen is located in the X direction. F :x F = x – (tan(90-θ-θ')*z).

[0221] Similarly, see Figure 7B On the YZ plane, the mouth is located at point P, with coordinates (y, z). Based on the principle of triangle similarity, we can obtain Equation 5: f / z = yl / y. yl represents the position of the preset facial feature (e.g., the mouth) on the first image, and the distance between it and the midline passing through the center of the first image and perpendicular to LR. From Equation 5, we can derive Equation 2: y = yl*z / f. Therefore, the coordinates y of point P can be obtained. The phone can calculate φ using Equation 6: φ = arctan(y / z). φ represents the angle between PL and PT when the user's face is directly facing the screen.

[0222] Then, the phone can calculate the Y-coordinate of the intersection point F between the extended line of the blowing direction and the plane where the screen is located. F y F =y–(tan(90-φ-φ')*z).

[0223] Among them, x F and y F The indicated position is the first point on the plane where the airflow reaches the screen, and this first position is relative to the origin of the coordinate system, i.e., the position of the camera (L). The phone can determine whether this first position is on the screen based on the screen's position relative to the camera. For example... Figure 8 As shown, if the first position is on the screen, then the first position is the target position A where the airflow reaches the screen.

[0224] The mobile phone determines the position of the airflow reaching the screen based on each set of target images captured in real time, thus forming an airflow trajectory. The directional trend of the airflow trajectory on the screen can be called the airflow direction. In some embodiments, the mobile phone can also display the current airflow direction or the trajectory formed by the airflow reaching the target position on the screen, allowing users to intuitively see the blowing direction, thereby preset the blowing response effect and improving the fun and user experience. For example, when the airflow trajectory is as follows... Figure 9A When the dashed line 901 is shown, the airflow direction is from bottom to top; when the airflow trajectory is as shown... Figure 9B As shown by the dashed line 902, the airflow direction is from left to right. Another example is... Figure 9C As shown, a mouth shape can be displayed on the screen, with the direction the mouth faces indicating the airflow direction. Another example is... Figure 9D As shown, a hand shape can be displayed on the screen, with the direction of the palm indicating the direction of airflow.

[0225] It should be noted that since most electronic devices such as mobile phones already have optical imaging devices such as cameras, mobile phones can use existing optical imaging devices in combination with infrared imaging devices to determine the three-dimensional information between the face and the phone (e.g., between the mouth and the camera) when blowing air. Then, based on this three-dimensional information, the position of the airflow on the screen and the direction of the airflow can be determined, without the need to set up a dedicated 3D camera. Thus, the multi-dimensional features of blowing air can be obtained with fewer cameras.

[0226] Further, compared with the microphone detection scheme in the prior art which can only detect whether the user blows air, but cannot detect other characteristics of blowing air, the blowing air detection scheme provided in the embodiments of the present application can also detect more abundant multi-dimensional characteristics such as blowing air direction, blowing air position, air flow position on the screen, blowing air intensity, blowing air time length, blowing air frequency, blowing air mode or blowing air rhythm. Therefore, the mobile phone can perform complex response operations or input control according to the more abundant multi-dimensional characteristics, enhance the human-computer interaction capability, and improve the entertainment and user experience.

[0227] In some other embodiments, the mobile phone can also detect whether the user blows air and multi-dimensional characteristics of blowing air through other sensors other than the camera and the infrared thermal imager. For example, the outer surface of the screen of the mobile phone or the inside of the screen can be provided with one or more thermosensitive elements. When the user blows air at the screen, the temperature of the corresponding position of the screen will change after the air flow reaches the screen. Similarly, the mobile phone can also detect whether the user blows air and multi-dimensional characteristics of blowing air according to the area change rate of the high-temperature region detected by the thermosensitive element. When the mobile phone determines that the user blows air, the position of the screen where the temperature changes is the position of the air flow on the screen.

[0228] For another example, the outer surface of the screen of the mobile phone or the inside of the screen can be provided with one or more pressure sensors. When the user blows air at the screen, the pressure of the corresponding position of the screen will change after the air flow reaches the screen. Similarly, the mobile phone can also detect whether the user blows air and multi-dimensional characteristics of blowing air according to the area change rate of the high-temperature region detected by the pressure sensor. When the mobile phone determines that the user blows air, the position of the screen where the pressure changes is the position of the air flow on the screen.

[0229] The mobile phone can respond according to the characteristics of the blowing air behavior. The characteristics of the blowing air behavior can include the target position, i.e., the position of the air flow on the screen. For example, the mobile phone responds to the target object displayed at the target position on the screen. For example, the target object at the target position on the screen is a ping-pong ball, and the ping-pong ball at the target position bounces once in response to the blowing air operation of the user.

[0230] In addition to the target position, the characteristics of the blowing air behavior can also include the air flow direction, air flow trajectory, blowing air time length, blowing air intensity, blowing air mode, blowing air frequency, blowing air rhythm, blowing air direction, or blowing air distance (i.e., the depth between the mouth and the screen when blowing air).

[0231] The mobile phone can determine the characteristics of the user's current blowing behavior according to the first image and / or the second image collected, and thus execute a corresponding response operation. In some embodiments, the mobile phone is preconfigured with a corresponding relationship between the response operation and different characteristics of the blowing behavior. In other embodiments, the corresponding relationship between the response operation and different characteristics of the blowing behavior can be set by the user through a system setting interface or through a dedicated application (App).

[0232] For example, one or more of the characteristics of the blowing behavior, the size, color, shape, transparency, type, material, number, moving direction, moving distance, rotation angle, action range, or response times of the target object are different.

[0233] For example, the target object displayed at the target position is a pig as shown in Figure 9D The greater the blowing strength, the greater the distance the pig moves; the smaller the blowing strength, the smaller the distance the pig moves. The blowing direction is different, and the moving direction of the pig is different. Single blowing, the pig moves the position. Continuous blowing for multiple times, the pig jumps, and the more the number of jumps. The blowing duration is different, and the material of the pig is different. For example, the blowing duration is short, and the pig is a rubber pig; the blowing duration is long, and the pig is a steel pig.

[0234] For another example, the target object displayed at the target position can be different. For example, the blowing strength is small, and the target object displayed at the target position is a lizard; the blowing strength is large, and the target object displayed at the target position is a tyrannosaurus.

[0235] For another example, the target object displayed at the target position is a balloon in the lower left corner. Figure 10A The state of the balloon before blowing. As shown in Figure 10B The longer the blowing duration, the greater the blowing strength, and the greater the volume of the balloon in the lower left corner grows. As shown in Figure 10C The shorter the blowing duration, the smaller the blowing strength, and the smaller the volume of the balloon grows. The blowing direction is different, and the direction of the balloon floating is different. Each blowing, the balloon changes a color. Continuous blowing for two times, the balloon color is red. Continuous blowing for three times, the balloon color is green.

[0236] For another example, the target object displayed at the target position is a gun, and the rotation angle of the gun barrel is different when the blowing strength is different.

[0237] For another example, the mobile phone scrolls the page according to the characteristics of the blowing behavior; and the scrolling direction and / or the scrolling amplitude of the page is different when the characteristics of the blowing behavior are different. Alternatively, the mobile phone turns the page according to the characteristics of the blowing behavior; and the order of turning the page forward or backward and / or the number of turning the page is different when the characteristics of the blowing behavior are different. The mobile phone updates the interface content according to the characteristics of the blowing behavior; and the updating effect of the interface is different when the characteristics of the blowing behavior are different.

[0238] Exemplarily, Figure 11A the interface displayed by the mobile phone before blowing, Figure 11B and Figure 11C the interface displayed by the mobile phone after blowing. Figure 11B corresponding to a large blowing strength, Figure 11C corresponding to a small blowing strength.

[0239] For another example, the mobile phone performs a corresponding touch operation according to the feature of the blowing behavior. For example, the touch operation can be a variety of basic touch input operations on the screen. The type and / or operation mode of the input operation are different according to the feature. Exemplarily, the type can include a click operation, a sliding operation, or a dragging operation, etc. The operation mode of the click operation can include one or more of a click position, a click number (such as single click, double click, or multiple clicks, etc.), a click strength (such as light touch or heavy press, etc.), or a press duration (such as heavy press), etc. The operation mode of the sliding operation includes one or more of a sliding position, a sliding direction, or a sliding distance, etc. The operation mode of the dragging operation includes one or more of a dragged object, a dragging direction, or a dragging distance, etc.

[0240] Exemplarily, a large blowing strength corresponds to a heavy press, and a small blowing strength corresponds to a light touch. For another example, a light blowing into the middle area of the screen can correspond to a long press operation; a forceful blowing into the middle area of the screen can correspond to a single click operation; a continuous blowing twice within a preset duration can correspond to a double click operation; a continuous blowing from the bottom of the screen to the top of the screen can correspond to an upward sliding operation, etc.

[0241] In some embodiments, the mobile phone can display an operation mark on the screen to simulate the touch input operation of the user on the screen. For example, when the mobile phone displays a browser interface and detects a continuous blowing from the bottom of the screen to the top of the screen, the mobile phone displays a hand shape moving upward and an arrow as shown in Figure 12A to simulate the touch input operation of the user on the screen. When the mobile phone detects a blowing into the middle area of the screen, the mobile phone displays a hand shape as shown in Figure 12B to simulate the touch input operation of the user on the screen. When the mobile phone detects a continuous blowing twice within a preset duration, the mobile phone displays a hand shape as shown in

[0242] The corresponding relationship between the response operation and the feature of the blowing behavior of the user can be the same or different for different Apps on the mobile phone. The mobile phone performs a response operation corresponding to the feature of the blowing behavior of the user according to the currently running App.

[0243] For example, in a mirror application, the user blows gently, and the fog on the mirror surface is small, and the mirror is more blurred; the user blows hard, and the fog on the mirror surface is large, and the blurring degree of the mirror is larger. In a reader, the user blows air from right to left, corresponding to the operation of turning back a page. In a gallery, the user blows air from left to right, corresponding to switching the previous picture; the user blows twice in succession, corresponding to zooming in or out of the picture; the user blows air continuously in the middle area of the screen, corresponding to exiting the gallery and returning to the desktop.

[0244] In a Chinese ink painting application, the mobile phone can control the direction, distance, and position after movement of the ink flow on the screen according to the direction and intensity of the user's blowing, so as to simulate the effect of the ink flowing on the paper in the actual scene and draw creative Chinese ink paintings.

[0245] The mobile phone can also respond through sound. And one or more of the tune, pitch, timbre, scale, or channel of the sound is different according to the characteristics of the blowing behavior. For example, in a wind chime playing application, the mobile phone plays music according to the blowing position, and shakes the wind chime at the position to play music; the mobile phone continuously shakes all the wind chimes to play music within a preset time period after determining that the user blows twice in succession. For another example, the blowing intensity is small, and a single instrument plays; the blowing intensity is large, and multiple instruments play symphonic music.

[0246] In some game applications, the mobile phone determines that the distance between the mouth and the screen is far when blowing, and the target object in the game jumps with a small amplitude; the mobile phone determines that the distance between the mouth and the screen is close when blowing, and the target object in the game jumps with a large amplitude.

[0247] In some social applications, the mobile phone can notify friends through the social application APP when detecting that the user blows, so that the friend's mobile phone screen appears interactive effects such as wind, rain, hail, falling petals, cake, screen cracking, and the like. For example, when the user blows gently, the friend's screen can appear a small amount of cracks; when the user blows hard, the friend's entire screen can be completely cracked.

[0248] In some augmented reality (AR) applications, the user blows air in combination with AR virtual technology, and can also achieve special effects such as shooting the user spewing fire and water.

[0249] For another example, the mobile phone can also respond through light. And one or more of the color of the light, the time period of lighting of the light, the pattern projected by the light, the number of light beams, or the flashing frequency of the light is different according to the characteristics of the blowing behavior. For example, blowing once, a single color of light flashes; blowing twice in succession, multiple colors of light interweave to form a colorful effect.

[0250] For another example, the mobile phone can also respond by vibrating. And the characteristics of the blowing behavior are different, one or more of the frequency, amplitude, or duration of the vibration are also different. For example, the blowing strength is small, the mobile phone vibrates once; the blowing strength is large, the mobile phone vibrates continuously according to a certain rhythm.

[0251] For another example, the mobile phone responds through a variable hardware mechanism. The variable hardware mechanism can be at the target position, or at the position corresponding to the target position (for example, other positions in the same horizontal direction as the target position), or at any position of the mobile phone. And the characteristics of the blowing behavior are different, one or more of the extension direction, extension degree, rotation direction, rotation degree, shape, size, material, or color of the variable mechanism are also different.

[0252] For example, the variable mechanism can be a liftable device (such as a liftable camera). The greater the blowing strength, the greater the amplitude of the liftable camera lifting / descending. If the blowing direction is upward, the liftable camera rises upward; if the blowing direction is downward, the liftable camera descends downward.

[0253] It can be seen that in the embodiments of the present application, the mobile phone can obtain the multi-dimensional features of the user according to the images collected by the camera and the infrared thermal imager, and thus perform different response operations or multiple input controls according to the multi-dimensional features, enhance the human-computer interaction capability, and improve the interest and user experience. The microphone detection scheme in the prior art can only detect whether the user blows or not, cannot map more input operations according to the user blowing, and cannot perform more function controls, so the function is relatively single and the user experience is poor.

[0254] Moreover, in the embodiments of the present application, performing multiple response operations or different input controls through blowing behavior can be used as an emergency input method, and can also facilitate human-computer interaction for special groups or users whose fingers are inconvenient to touch the screen.

[0255] The above is described taking the mobile phone as an example. The blowing detection method provided in the above embodiments of the present application can also be applied to other types of mobile phones, which will not be described in detail here.

[0256] In addition, when the screen of the mobile phone is large, multiple users can also interact through blowing for multi-player game. Alternatively, multiple mobile phones can also be used in cooperation, the game interface can be projected onto a large-screen device, and multiple users can blow through their own mobile phones, and the large-screen device can display the game interaction of multiple users. Thus, the game interest can be improved, and the user experience can be improved.

[0257] In combination with the above drawings Figures 2-12BAccording to the description of the embodiments of the present application, the embodiments of the present application further provide a blowing detection method, which can be applied to an electronic device with a first image acquisition device and a second image acquisition device. Referring to Figure 13 , the method can include:

[0258] 1301. The electronic device acquires a first image and a second image, the second image being a thermal image.

[0259] The first image is an image of the user's face taken by the first image acquisition device, and the second image is a thermal image of the user's face taken by the second image acquisition device. For example, the electronic device can be a mobile phone with the structure as shown in Figure 2 , the first image acquisition device can be a camera as shown in Figure 2 , and the second image acquisition device can be an infrared thermal imager as shown in Figure 2 .

[0260] 1302. The electronic device identifies a blowing behavior of the user according to the second image.

[0261] The electronic device can identify whether the user blows, etc. blowing behavior according to the thermal image taken of the user's face. In some embodiments, the electronic device can include a screen, and the electronic device can identify the blowing operation of the user on the screen according to the second image.

[0262] 1303. The electronic device identifies the orientation of the user's face according to the first image and / or the second image.

[0263] 1304. The electronic device determines the blowing direction of the user according to the orientation of the user's face.

[0264] For example, the electronic device can determine the angles θ' and φ' of the face deflection of the user when blowing according to the orientation of the user's face, as shown in Figure 7A , and the direction defined by the angles of the face deflection is the blowing direction of the user.

[0265] 1305. The electronic device determines the three-dimensional information of the user's face relative to the electronic device according to the first image and the second image.

[0266] For example, the three-dimensional information of the user's face relative to the electronic device is specifically the three-dimensional information of the mouth relative to the camera. The electronic device determines the three-dimensional information based on the binocular distance measuring principle as shown in Figure 7A and Figure 7B .

[0267] 1306. The electronic device determines the target position of the blowing airflow reaching the electronic device according to the blowing direction and the three-dimensional information.

[0268] For example, the electronic device can determine the target position of the blowing airflow reaching the electronic device according to Figures 7A-8The process shown determines the target position A. After obtaining the target position, the electronic device can determine the air flow direction on the electronic device according to multiple target positions.

[0269] 1307. The electronic device performs a response operation according to a feature of the blowing behavior, which includes the target position.

[0270] In addition, the feature of the blowing behavior can also include the air flow direction, the air flow trajectory, the blowing duration, the blowing intensity, the blowing mode, the blowing frequency, the blowing rhythm, the blowing direction, or the blowing distance, etc. For example, the electronic device can perform a response operation according to the feature of the blowing behavior, which can be referred to the above-mentioned Figures 10A-12B .

[0271] In this scheme, the electronic device can determine the three-dimensional information between the face and the electronic device during blowing by using the commonly used first image acquisition device in combination with the infrared camera (i.e. the second image acquisition device), and then determine the position of the air flow on the screen and the direction of the air flow according to the three-dimensional information, without the need to specially set a 3D camera, so that the blowing behavior of the user can be recognized by using fewer cameras.

[0272] In addition, the electronic device can also obtain the multi-dimensional feature of the blowing behavior of the user according to the thermal image collected by the infrared thermal imager, so as to perform different response operations or multiple input controls according to the multi-dimensional feature, enhance the human-computer interaction ability, and improve the interest and user experience.

[0273] In some embodiments, the above-mentioned step 1302 can specifically include:

[0274] 1302a. The electronic device identifies the air flow images on multiple second images.

[0275] 1302b. The electronic device identifies the blowing behavior of the user according to the area change rate of the air flow images between adjacent second images in the multiple second images.

[0276] Wherein, the area change rate dS is: dS = (S2-S1) / T. S1 represents the area of the air flow image on the previous second image in the two second images collected in sequence; S2 represents the area of the air flow image on the next second image; and T represents the collection interval of the two adjacent second images.

[0277] Referring to Figure 14 , the step 1302b can specifically include:

[0278] 1401. If the average value of the area change rate of the air flow images between adjacent second images in the multiple second images collected in the first detection period is greater than or equal to the first preset value, the electronic device determines that the blowing operation of the user is detected.

[0279] If the average of the area change rate of the airflow image in the first detection period is greater than or equal to the second preset value, which is greater than the first preset value, the blowing strength of the blowing operation is large. If the average of the area change rate of the airflow image in the first detection period is greater than or equal to the first preset value and less than the second preset value, the blowing strength of the blowing operation is small.

[0280] On the basis of step 1401, the above step 1302b can further include:

[0281] 1402. After the electronic device determines that the blowing operation of the user is detected, if the average of the area change rate of the airflow image between adjacent second images in the plurality of second images collected in the second detection period is less than a third preset value, which is less than or equal to the first preset value, the electronic device determines that the user stops blowing.

[0282] The time length from when the electronic device determines that the blowing operation of the user is detected to when the electronic device determines that the user stops blowing is the blowing time length of the blowing operation.

[0283] In some other embodiments, the step 1302b can specifically include:

[0284] 1403. If the electronic device detects that the area change rate of the airflow image between two adjacent collected second images is greater than or equal to the first preset value, the electronic device determines that the blowing operation of the user is detected.

[0285] 1404. If the electronic device detects that the area change rate of the airflow image between two adjacent collected second images is less than a third preset value, which is less than or equal to the first preset value, the electronic device determines that the user stops blowing.

[0286] On the basis of the above steps 1401-1404, the above step 1302b can further include:

[0287] 1405. The electronic device determines the blowing and stopping blowing conditions according to the change of the area change rate of the airflow image between adjacent second images in the plurality of second images.

[0288] 1406. The electronic device determines the blowing mode of the user according to the blowing and stopping blowing conditions, the blowing mode including single blowing or continuous multiple blowings within a preset time length.

[0289] Embodiments of the present application also provide an electronic device, which can include an acquisition unit, an identification unit, a determination unit, an execution unit, and a display unit, etc. These units can perform the steps in the above embodiments to implement the blowing detection method.

[0290] The embodiment of the present application further provides an electronic device, comprising one or more processors, a memory, a first image acquisition device, a second image acquisition device, and one or more computer programs. The first image acquisition device is configured to acquire an image of a user's face. The second image acquisition device is configured to acquire a thermal image of the user's face. The one or more computer programs are stored in the memory, and the one or more computer programs comprise instructions. When the instructions are executed by the one or more processors, the electronic device performs the steps in the above embodiment to implement the blowing detection method.

[0291] For example, when the electronic device is a device with the structure shown in Figure 2 the processor in the electronic device can be the processor 110 in Figure 2 the memory in the electronic device can be the internal memory 121 in Figure 2 the first image acquisition device in the electronic device can be the camera 193 in Figure 2 the second image acquisition device in the electronic device can be the infrared thermal imager 196 in Figure 2

[0292] The embodiment of the present application further provides a computer storage medium, which stores computer instructions. When the computer instructions are run on an electronic device, the electronic device performs the steps of the above related method to implement the blowing detection method in the above embodiment.

[0293] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computer performs the above related steps to implement the blowing detection method in the above embodiment.

[0294] In addition, the embodiment of the present application further provides a device, which can be a chip. The chip can comprise a processor and a memory. The memory stores instructions. When the instructions are executed by the processor, the chip performs the above related steps to implement the blowing detection method in the above embodiment.

[0295] Alternatively, the device can be a chip system applied to an electronic device with a first image acquisition device and a second image acquisition device. The chip system comprises one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected through a circuit. The interface circuit is configured to receive signals from a memory of the electronic device and send signals to the processor. The signals comprise computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device performs the above related steps to implement the blowing detection method in the above embodiment.

[0296] ​In addition, the embodiment of the present application further provides a device, which can be a component or a module. The device can include a processor and a memory connected to each other. The memory is used to store computer-executed instructions. When the device is running, the processor can execute the computer-executed instructions stored in the memory, so that the chip executes the blowing detection method in the above-mentioned method embodiments.

[0297] The electronic device, the chip, the computer storage medium, the computer program product or the chip provided by the embodiment of the present application are all used to execute the corresponding method provided above, and thus the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method provided above, which will not be described here.

[0298] Through the description of the above embodiments, those skilled in the art can understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0299] In several embodiments provided by the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only schematic, and the division of the modules or units is only a logical function division, and there can be another division way in actual implementation. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0300] The units described as separate components can or can not be physically separate, and the components displayed as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0301] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of software functional unit.

[0302] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, including a plurality of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0303] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting air blowing, applied to an electronic device, the electronic device comprising a first image acquisition device and a second image acquisition device, characterized in that, The method comprises: The electronic device collects a first image and a plurality of second images; the first image is an image captured by the first image collection device for the user's face, and the plurality of second images are thermal images captured by the second image collection device for the user's face; The electronic device identifies the user's blowing behavior according to the area change rate of the airflow image between adjacent second images in the plurality of second images; The electronic device identifies the orientation of the user's face according to the first image and / or the second image; The electronic device determines the blowing direction of the user according to the orientation of the user's face; the blowing direction is a direction defined by the angle of the face deflection when the user blows; The electronic device determines the three-dimensional information of the user's face relative to the electronic device according to the first image and the second image; the three-dimensional information includes the three-dimensional information of the user's mouth relative to the first image collection device; The electronic device determines the target position of the airflow reaching the electronic device according to the blowing direction and the three-dimensional information; The electronic device performs a response operation according to the characteristics of the blowing behavior, and the characteristics include the target position.

2. The method of claim 1, wherein, The method further comprises: The electronic device determines the airflow direction on the electronic device according to a plurality of target positions.

3. The method of claim 1, wherein, The area change rate dS is: dS = (S2-S1) / T; Wherein, S1 represents the area of the airflow image on the previous second image in the two adjacent collected second images; S2 represents the area of the airflow image on the next second image; T represents the collection interval of the two adjacent second images.

4. The method of claim 3, wherein, The electronic device identifies the user's blowing behavior according to the area change rate of the airflow image between adjacent second images in a plurality of second images, comprising: If the average value of the area change rate of the airflow image between adjacent second images in a plurality of second images collected in a first detection period is greater than or equal to a first preset value, the electronic device determines that the user's blowing operation is detected.

5. The method of claim 4, wherein, If the average value of the area change rate of the airflow image in the first detection period is greater than or equal to a second preset value, the second preset value is greater than the first preset value, the blowing strength of the blowing operation is large; If the average value of the area change rate of the airflow image in the first detection period is greater than or equal to the first preset value and less than the second preset value, the blowing strength of the blowing operation is small.

6. The method of claim 4, wherein, The electronic device identifies the user's blowing behavior according to the area change rate of the airflow image between adjacent second images in a plurality of second images, further comprising: After the electronic device determines that the user's blowing operation is detected, if the average value of the area change rate of the airflow image between adjacent second images in a plurality of second images collected in a second detection period is less than a third preset value, the third preset value is less than or equal to the first preset value, the electronic device determines that the user stops blowing; The length of time from when the electronic device determines that the blowing operation of the user is detected to when the electronic device determines that the blowing of the user is stopped is a blowing time length of the blowing operation.

7. The method of claim 3, wherein, The electronic device identifies the blowing behavior of the user according to the area change rate of the airflow image between adjacent second images in a plurality of second images, including: If the electronic device detects that the area change rate of the airflow image between two adjacent collected second images is greater than or equal to a first preset value, the electronic device determines that the blowing operation of the user is detected. If the electronic device detects that the area change rate of the airflow image between two adjacent collected second images is less than a third preset value, the third preset value is less than or equal to the first preset value, the electronic device determines that the user stops blowing.

8. The method according to any one of claims 4-7, characterized in that, The electronic device identifies the blowing behavior of the user according to the area change rate of the airflow image between adjacent second images in a plurality of second images, including: The electronic device determines the blowing and stopping blowing according to the change of the area change rate of the airflow image between adjacent second images in a plurality of second images. The electronic device determines the blowing mode of the user according to the blowing and stopping blowing, the blowing mode including single blowing or continuous multiple times blowing within a preset time length.

9. The method according to any one of claims 1 to 7, characterized in that, The electronic device includes a screen, and the electronic device identifies the blowing behavior of the user according to the second image, including: The electronic device identifies the blowing behavior of the user for the screen according to the second image.

10. The method according to any one of claims 1 to 7, characterized in that, The characteristics of the blowing behavior further include one or more of blowing direction, blowing distance, airflow direction, blowing strength, blowing time length, blowing times, blowing frequency, blowing rhythm, or blowing mode.

11. The method of claim 10, wherein, The electronic device performs a response operation according to the characteristics of the blowing behavior, including: The electronic device performs a response operation according to the characteristics of the blowing behavior for a target object displayed at the target position; Wherein, the characteristics are different, and one or more of the size, color, shape, transparency, type, material, number, moving direction, moving distance, rotation angle, action range, or response times of the target object are also different.

12. The method of claim 10, wherein, The electronic device performs a response operation according to the characteristics of the blowing behavior, including: The electronic device performs a corresponding touch operation according to the characteristics of the blowing behavior; Wherein, the characteristics are different, and the type and / or operation mode of the touch operation are different; the type includes click operation, sliding operation, or dragging operation; The operation mode of the click operation includes one or more of click position, click times, click force, or press time length; The operation mode of the sliding operation includes one or more of sliding position, sliding direction, or sliding distance; The operation mode of the dragging operation includes one or more of dragging object, dragging direction, or dragging distance.

13. The method of claim 10, wherein, The electronic device performs a response operation according to the characteristics of the blowing behavior, including: The electronic device scrolls a page according to a characteristic of the blowing behavior; and the characteristic is different, the scrolling direction and / or the scrolling range of the page are different; Or, the electronic device turns a page according to a characteristic of the blowing behavior; and the characteristic is different, the order of turning the page forward or backward is different and / or the number of turning the page is different; The electronic device updates the interface content according to the characteristic of the blowing behavior; and the characteristic is different, the updating effect of the interface is different.

14. The method of claim 10, wherein, The electronic device responds according to the characteristic of the blowing behavior, including: The electronic device responds through sound, and the characteristic is different, one or more of the tune, pitch, timbre, scale, or channel of the sound are also different; Or, the electronic device responds through light, and the characteristic is different, one or more of the color, light-on duration, light projection pattern, number of light beams, or flicker frequency of the light are also different; Or, the electronic device responds through vibration, and the characteristic is different, one or more of the frequency, amplitude, or duration of the vibration are also different.

15. The method of claim 10, wherein, The electronic device responds according to the characteristic of the blowing behavior, including: The electronic device responds through a variable mechanism, and the characteristic is different, one or more of the extension direction, extension degree, rotation direction, rotation degree, shape, size, material, or color of the variable mechanism are also different.

16. An electronic device, comprising: Including: One or more processors, memories, first image acquisition devices, and second image acquisition devices; The first image acquisition device is configured to acquire an image of a user's face; The second image acquisition device is configured to acquire a thermal image of the user's face; The memory stores code; when the code is executed by the one or more processors, the electronic device executes the blowing detection method according to any one of claims 1-15.

17. A computer storage medium, comprising, The computer program product includes computer instructions that, when executed on a computer, cause the computer to execute the blowing detection method according to any one of claims 1-15.

18. A computer program product, characterised in that, When the computer program product is executed on a computer, the computer executes the blowing detection method according to any one of claims 1-15.

19. A chip system, characterized by The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a circuit; the interface circuit is configured to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the blowing detection method according to any one of claims 1-15.

Images