Identification method and electronic device

Abstract

The embodiment of the present application provides a recognition method, which is applied to electronic devices, and the method includes: obtaining a first image and a second image, the first image is an image taken by an optical camera, the second image is an image taken by an event camera, and the second image is determined according to the brightness change of the pixels of the moving target object; obtaining the key point information of N parts of the target object according to the first image; determining the first moving part among the N parts according to the key point information and the grayscale value of the second image, and the grayscale value of the pixel of the part with a greater movement frequency in the second image is greater. The use of the embodiment of the present application can improve the recognition accuracy of the moving part, and the user can interact with the electronic device without external devices such as keyboards and wearable devices, which is more convenient.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to an identification method and electronic equipment. Background Art

[0002] At present, electronic devices such as augmented reality (AR) glasses and virtual reality (VR) glasses can capture images of users through configured optical cameras, and identify the user's moving parts (such as fingers performing tapping actions) based on the image, so as to obtain information input by the user (such as text information). No external devices such as keyboards, wearable devices, and cameras with third-party perspectives are required, which greatly facilitates the use of users. However, the captured image may have the problem of motion blur, resulting in inaccurate recognition results. For example, when a user wears VR glasses, text can be input through a virtual keyboard in the virtual world. VR glasses can capture images of the user's hands through their own optical cameras, and obtain the coordinates of the key points of the user's hands based on the image, and then identify the motion information such as the tapping fingers based on the coordinates, and finally obtain the text currently input by the user in combination with the input text. However, when the user's hand and background colors are similar and the finger tapping speed is too fast, the captured image has the problem of motion blur, resulting in low accuracy of the obtained hand key points and inaccurate text. Summary of the invention

[0003] The embodiments of the present application disclose a recognition method and an electronic device, which can interact with the electronic device without the need for external devices such as a keyboard and a wearable device, while improving the recognition accuracy of moving parts and obtaining more accurate input information.

[0004] In a first aspect, an embodiment of the present application provides a recognition method, which is applied to an electronic device, and the method includes: the electronic device acquires a first image and a second image, the first image is an image taken by an optical camera, the second image is an image taken by an event camera, and the second image is determined based on the brightness change of pixels of a moving target object; the electronic device acquires key point information of N parts of the target object based on the first image, where N is a positive integer; the electronic device determines a first moving part among the N parts based on the key point information and the grayscale value of the second image, and a part with a greater movement frequency has a greater grayscale value of the pixel in the second image.

[0005] In the embodiment of the present application, the electronic device can combine the optical camera and the event camera to determine the moving part of the target object. Even if the first image has problems such as motion blur, overexposure, dim light, etc., more accurate motion information can be obtained by combining with the second image. At the same time, the user can interact with the electronic device without external devices such as keyboards, wearable devices, and third-party cameras, which enhances the interactive ability of the electronic device and makes it more convenient for users to use.

[0006] In a possible implementation, the electronic device obtains key point information of N parts of the target object based on the first image, including: the electronic device fuses the first image and the second image to obtain a third image; the electronic device obtains key point information of the N parts based on the third image.

[0007] In an embodiment of the present application, the electronic device can combine an optical camera and an event camera to realize key point recognition, reducing the impact of the image quality of the first image on the key point detection accuracy in cases such as motion blur and similar color and texture of the object and background, and the motion information obtained based on the key points is also more accurate and robust.

[0008] In a possible implementation, before the electronic device fuses the first image and the second image to obtain the third image, the method further includes: the electronic device determines a first weight of the first image and a second weight of the second image based on parameters of the first image; the parameters of the first image include at least one of the following: distribution, mean, and standard deviation of a grayscale histogram; the electronic device fuses the first image and the second image to obtain the third image, including: the electronic device fuses the first image and the second image based on the first weight and the second weight to obtain the third image.

[0009] In a possible implementation, the electronic device determines the first weight of the first image and the second weight of the second image according to the parameters of the first image, including: when a preset condition is met, the electronic device sets the first weight and the second weight to a first preset value and a second preset value, respectively, wherein the first preset value is less than the second preset value; the preset condition includes at least one of the following: the distribution of the grayscale histogram of the first image is concentrated within a fixed interval, the mean of the first image is greater than a first threshold, the mean of the first image is less than a second threshold, and the standard deviation of the first image is less than a third threshold, wherein the first threshold is greater than the second threshold.

[0010] In the embodiment of the present application, when the imaging effect of the first image is poor (for example, when the preset conditions are met), the electronic device can reduce the weight of the first image when acquiring the third image, and increase the weight of the second image. The third image is used to obtain key points, so the influence of the quality of the first image on the detection accuracy of key points can be reduced in cases such as overexposure of the image and dim light, and the motion information obtained based on the key points is more accurate.

[0011] In a possible implementation, the electronic device obtains the key point information of the N parts based on the third image, including: the electronic device identifies the target area where the N parts are located in the third image; the electronic device identifies the key point information of the N parts in the target area.

[0012] In an embodiment of the present application, the electronic device can first obtain the target area where the N parts of the target object are located, and then perform key point detection based on the target area. There is no need to perform key point detection in areas outside the target area, thereby avoiding unnecessary processing procedures, reducing processing pressure, and improving usability.

[0013] In a possible implementation, the electronic device determines the first moving part among the N parts according to the key point information and the grayscale value of the second image, including: the electronic device determines M parts from the N parts according to the key point information, M is less than or equal to N, and M is a positive integer; the key point information includes the coordinates of at least one key point on the N parts; the electronic device determines the first part from the M parts according to the grayscale value of the second image, the grayscale value of the pixel of the first part in the second image is greater than a preset grayscale threshold, or the grayscale value of the pixel of the first part in the second image is greater than the grayscale value of the pixel of other parts among the M parts in the second image.

[0014] In an embodiment of the present application, the electronic device can first obtain motion information (i.e., the M parts of the motion mentioned above) based on the key point information, and then filter and process the motion information according to the grayscale value of the second image, so as to obtain a more accurate motion part: the first part, with higher recognition accuracy.

[0015] In one possible implementation, the difference between the coordinates of the key points on the above-mentioned M parts at the first moment and the coordinates at the second moment is greater than the first preset difference, and the above-mentioned first moment and the above-mentioned second moment are different; or, the difference between the coordinates of the key points on the above-mentioned M parts and the preset coordinates is greater than the second preset difference.

[0016] Exemplarily, or alternatively, the difference between the coordinates of the key points on the above-mentioned M parts at the first moment and the preset coordinates is a first difference, the difference between the coordinates of the key points on the above-mentioned M parts at the second moment and the preset coordinates is a second difference, and the difference between the first difference and the second difference is greater than a third preset difference.

[0017] For example, the preset coordinates are the coordinates of a key on a virtual keyboard displayed by the electronic device that corresponds to the key point.

[0018] In the embodiment of the present application, there are various ways to determine the M parts that may move. Even if only one frame of the first image is acquired, the M parts that may move can be determined by the difference between the coordinates of the key points and the coordinates of the keys on the virtual keyboard, thereby reducing processing delays and providing a wider range of application scenarios.

[0019] In a possible implementation, the first part is used by the electronic device to determine first information input by the target object through the first part.

[0020] In the embodiment of the present application, the electronic device can combine the optical camera and the event camera to improve the detection accuracy of the moving parts, and the moving parts are used to obtain the information input by the target object, so the recognition accuracy of the input information can be improved. At the same time, the user can input information to the electronic device without external devices such as keyboards, wearable devices, and cameras with third-party perspectives. Even if the user is in a mobile scene, the electronic device can be used normally, which is more convenient.

[0021] In a possible implementation, the method further includes: the electronic device determines Q pieces of information based on the first part, Q is a positive integer, and the Q pieces of information include the first information; the electronic device determines the first information from the Q pieces of information based on second information, and the second information is information input by the target object before the target object inputs the first information.

[0022] In the embodiment of the present application, the electronic device can combine the second information input by the target object and the first moving part to guess the first information input by the user through the first part, thereby improving the recognition accuracy of the input information.

[0023] In a possible implementation, the first part is a plurality of parts.

[0024] In a possible implementation, the target object is a user, and the first part is a finger of the user.

[0025] In an embodiment of the present application, the electronic device can combine an optical camera and an event camera to realize the recognition of the user's moving fingers, and thereby recognize the information input by the user through the fingers based on the virtual keyboard. The input information obtained is more accurate and does not require a physical keyboard, which greatly facilitates user use.

[0026] In a second aspect, an embodiment of the present application provides an electronic device, which includes one or more memories and one or more processors; the one or more memories are used to store computer programs, the one or more processors are used to call the computer programs, and the computer programs include instructions. When the instructions are executed by the one or more processors, the electronic device executes the identification method provided by the first aspect or any one of the implementation methods of the first aspect.

[0027] In a possible implementation, the electronic device includes the event camera and the optical camera.

[0028] In a possible implementation, the electronic device is a virtual reality device, an augmented reality device, or a mixed reality device.

[0029] In a third aspect, an embodiment of the present application provides a computer storage medium, including a computer program, which includes instructions. When the instructions are executed on a processor, the identification method provided by the first aspect or any one of the implementations of the first aspect is implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the identification method provided by the first aspect or any one of the implementations of the first aspect.

[0031] In the fifth aspect, an embodiment of the present application provides a chip, which includes at least one processor and an interface circuit, and optionally, the chip also includes a memory; the above-mentioned memory, the above-mentioned interface circuit and the above-mentioned at least one processor are interconnected through lines, and a computer program is stored in the above-mentioned at least one memory; when the above-mentioned computer program is executed by the above-mentioned at least one processor, the identification method provided by the first aspect and any one of the implementation methods of the first aspect is implemented.

[0032] It can be understood that the electronic device provided in the second aspect, the computer storage medium provided in the third aspect, the computer program product provided in the fourth aspect, and the chip provided in the fifth aspect are all used to execute the recognition method provided in the first aspect and any one of the implementations of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the recognition method provided in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The following is an introduction to the drawings used in the embodiments of the present application.

[0034] Figure 1 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0035] Figure 2 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0036] Figure 3 It is a schematic diagram of a processing process provided by an embodiment of the present application;

[0037] Figure 4-Figure 5 is a schematic diagram of some key points on a virtual keyboard provided in an embodiment of the present application;

[0038] Figure 6 It is a flowchart of an identification method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. The terms used in the implementation methods of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0040] The embodiment of the present application provides a recognition method that can be applied to an electronic device. The electronic device can obtain a first image and a second image, wherein the first image is obtained by taking an optical camera and the second image is obtained by taking an event camera. Then, the electronic device can recognize the key point information of the target object (such as the key point coordinates of the user's hand) based on the first image and the second image, and combine the key point information, the second image, and optionally the second information already entered by the user to obtain the first information currently entered by the user.

[0041] Among them, the output of the event camera is the brightness change at the pixel level. That is to say, in the pixel array, when the brightness change of a certain pixel is greater than the preset brightness threshold, the pixel will generate an output, which can be called an "event". Therefore, when the object photographed by the event camera is not moving, the output of the event camera is a black image. When the object photographed by the event camera moves, causing the brightness of multiple pixels to change, the output of the event camera can be a moving object. The event camera has the advantages of no requirements for lighting conditions, low latency, and sensitivity to small and fast movements. Therefore, the second image is determined based on the brightness change of the pixels of the moving target object, and the second image includes the part of the moving target object.

[0042] This application combines the images output by the optical camera and the event camera to identify the moving parts of the target object, reducing the impact of image quality on the detection accuracy of key point information when motion blur occurs, and the recognition result (i.e. the moving part of the target object, and / or the first information) is more accurate. In addition, the user can input information to the electronic device and perform human-computer interaction without external devices such as keyboards, wearable devices, and cameras with third-party perspectives. Even in mobile scenarios, the user can use the electronic device normally, which is more convenient.

[0043] Unlike the output of an event camera, an optical camera can output a complete image consisting of multiple pixels, such as an RGB image. When the object and background color in the image output by the optical camera are similar, or the motion speed is too fast, the image has the problem of motion blur. In addition, the imaging effect of the optical camera is greatly affected by the lighting, and the image quality is poor when the lighting is too bright or too dark. Exemplarily, an electronic device can obtain multiple frames of images including the user's hand through an optical camera, and obtain the position information of the key points of the user's hand based on these multiple frames of images (for example, including the coordinates of the key points of the hand relative to the center of the optical camera, and the coordinates of the key points of the hand relative to the center of the hand). Then the electronic device identifies whether there is a gesture matching the position information in the preset gesture. If so, the electronic device determines that the gesture currently input by the user is a gesture matching the position information, and performs the operation corresponding to the gesture, such as selecting a word or deleting an input word. If there is no gesture matching the position information, the electronic device can identify the motion information of the finger (such as finger coordinates, amplitude, etc.) according to the position information, and obtain the candidate word with the highest probability through the Bayesian model, combined with the finger motion information and the language model of the word usage frequency, and confirm the candidate word as the text input by the current user. Alternatively, the electronic device can also identify the motion information of the finger based on a single frame image to obtain the text input by the current user. However, a single frame image can only reflect the scene at a certain moment, while the finger movement is time-series and belongs to a scene within a period of time. Therefore, the accuracy of identifying motion information through a single frame image is low. At the same time, when the image captured by the optical camera has motion blur and is greatly affected by the lighting, the detection accuracy of the key points of the hand is also low, and the recognition result (i.e., the text input by the current user) is also inaccurate. In addition, the gestures of different users may vary greatly. When performing gesture matching, it is likely to fail to match and other misrecognition and unrecognition situations, which depends on the quality of the preset gesture and has low usability.

[0044] The embodiment of the present application does not limit the form of the information input by the user, such as but not limited to text information, picture information, audio information, instruction information, etc.

[0045] The electronic device involved in the embodiments of the present application may be a wearable electronic device, such as a head-mounted electronic device, glasses, goggles, etc., and a user may wear the wearable electronic device to achieve different effects such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). Not limited to this, the electronic device may also be other electronic devices including optical cameras and event cameras, such as mobile phones, tablet computers, laptops, smart screens, smart TVs, headphones, and other devices.

[0046] The embodiments of the present application are introduced by taking the electronic device as a head-mounted electronic device as an example, but the embodiments of the present application are not limited to head-mounted electronic devices, and the electronic device may also be other devices.

[0047] See also Figure 1 , Figure 1 A schematic structural diagram of an electronic device 100 is exemplarily shown.

[0048] like Figure 1 As shown, the electronic device 100 may include a processor 110 , a memory 120 , a communication module 130 , a display screen 140 , a sensor module 150 , and a camera 160 , etc. The camera 160 may include an optical camera 161 and an event camera 162 .

[0049] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the electronic device 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some embodiments, the processor 110 can also be connected to other processing units to collaboratively execute the recognition method provided in this application.

[0051] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0052] The memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image capture function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as image data, text data, etc.), etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0053] The electronic device 100 may include a wireless communication function. In some embodiments, the communication module 130 may include a mobile communication module and a wireless communication module. The wireless communication function may be implemented by an antenna, a mobile communication module, a wireless communication module, a modem processor, and a baseband processor.

[0054] The antenna is used to transmit and receive electromagnetic wave signals. The electronic device 100 may include multiple antennas, each of which may be used to cover a single or multiple communication frequency bands. Different antennas may also be reused to improve the utilization of the antenna. For example, antenna 1 may be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna may be used in combination with a tuning switch.

[0055] The mobile communication module can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module may receive electromagnetic waves through an antenna, and perform filtering, amplification, and other processing on the received electromagnetic waves, and transmit them to a modulation and demodulation processor for demodulation. The mobile communication module may also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module may be arranged in the same device as at least some of the modules of the processor 110.

[0056] In some embodiments, the antenna of the electronic device 100 is coupled to the mobile communication module so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The above-mentioned GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS) and / or satellite based augmentation system (SBAS).

[0057] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a voice signal through an audio device (not limited to a speaker, etc.), or displays an image or video through a display screen 1100. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module or other functional modules.

[0058] The wireless communication module can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module can be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0059] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc.

[0060] When the electronic device 100 is mounted on the user's head, the user's eyes can see the image presented by the display screen 140 of the electronic device 100. When the display screen 140 is transparent, the user's eyes can see the physical object through the display screen 140, or the user's eyes can see the image displayed by another display device through the display screen 140.

[0061] The number of display screens 140 in the electronic device 100 may be two, corresponding to the two eyeballs of the user respectively. The contents displayed on the two display screens may be displayed independently. Different images may be displayed on the two display screens to enhance the stereoscopic effect of the image. In some embodiments, the number of display screens 140 in the electronic device 100 may also be one, corresponding to the two eyeballs of the user.

[0062] In some embodiments, the camera 160 may include an optical camera 161 and an event camera 162 .

[0063] The optical camera 161 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. The optical camera 161 includes, for example but not limited to, a monocular camera, a binocular camera, and a depth camera. Among them, the depth camera can measure the depth information of the object through methods such as structured light or time of flight (TOF).

[0064] The output of the event camera 162 is the brightness change at the pixel level. That is, in the pixel array, when the brightness change of a certain pixel is greater than the preset brightness threshold, the pixel will generate an output, which can be called an "event". When the object movement or light change in the shooting scene causes a large number of pixel changes, the event camera 162 can output a series of events, which can also be called an event stream. The amount of data in the event stream is much smaller than the data output by the optical camera 161.

[0065] In some embodiments, the electronic device 100 may include multiple cameras. Specifically, the electronic device 100 may include at least one optical camera 161 and at least one event camera 162. Figure 1 As shown, the electronic device 100 includes four optical cameras 161, which are installed on the side of the electronic device 100, two at the top and two at the bottom (the one at the bottom is not shown). The electronic device also includes two event cameras 162, which are installed between two display screens 140 on the electronic device 100, one at the top and one at the bottom (not shown). The camera is used to capture images and videos within the user's perspective in real time. The electronic device 100 can generate a virtual image based on the captured real-time images and videos, and display the virtual image through the display screen 140.

[0066] Understandably, Figure 1 The positions and numbers of the optical camera 161 and the event camera 162 shown in the figure on the electronic device 100 are only used to explain the embodiments of the present application and should not constitute a limitation.

[0067] In the present application, the electronic device 100 can capture a first image through an optical camera 161 and a second image through an event camera 162. The processor 110 can fuse the first image and the second image through a first algorithm to obtain a third image, and identify key point information in the third image (for example, the coordinates of 21 or 22 key points of the user's hand). The processor 110 can then combine the key point information, the second image, and optionally the second information already input by the user to obtain the first information currently input by the user. The processor 110 can determine to perform a corresponding operation based on the first information. For example, if the first information is text information input by the user, the processor 110 can display the first information after the second information through the display screen 140. Alternatively, the first information is instruction information, and the processor 110 can perform a corresponding operation (for example, a shutdown operation, a pause operation, etc.) in response to the instruction information.

[0068] In some embodiments, the electronic device 100 can also be connected to other devices (such as mobile phones, tablet computers, smart screens, etc.), and the electronic device 100 can obtain a first image and a second image from the other devices, wherein the first image is taken by the other device through an optical camera, and the second image is taken by the other device through an event camera.

[0069] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, a serial peripheral interface (SPI) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, and / or a universal serial bus (USB) interface, etc.

[0070] In some embodiments, the electronic device 100 can implement display functions through a GPU, a display screen 140, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0071] In some embodiments, the electronic device 100 can implement the shooting function through an ISP, a camera, a video codec, a GPU, a display screen 140, and an application processor. The ISP can be used to process data fed back by an optical camera. For example, when taking a photo with an optical camera, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera.

[0072] The sensor module 150 may include a plurality of sensors, such as a touch sensor, a pressure sensor, an ambient light sensor, an acceleration sensor, a gyroscope sensor, an infrared sensor, etc., but is not limited thereto, and may also include a microphone, an earphone, etc.

[0073] In some embodiments, the processor 110 can determine the virtual image displayed on the display screen 140 based on the static image or video image captured by the camera and the data obtained by the sensor module 150 (such as brightness, sound, etc.), thereby superimposing the virtual image on the real-world object.

[0074] See also Figure 2 , Figure 2 A schematic diagram exemplarily shows a scenario in which a user inputs text information.

[0075] like Figure 2 As shown, the user 200 wears the electronic device 100. The structure of the electronic device 100 can be seen in the above Figure 1 The structure shown. The user 200 can see the virtual interface 300 and the virtual keyboard 400 presented by the display screen 140 of the electronic device 100. The virtual interface 300 can display the user interface of the application on the electronic device 100, or the user interface of the application of other devices connected to the electronic device 100 (such as mobile phones, tablet computers, smart screens, etc.). The virtual keyboard 400 can have the same structure as the physical keyboard. The user 200 can input information through the virtual keyboard 400, and the information can be presented on the virtual interface 300.

[0076] Exemplarily, it is assumed that the virtual interface 300 has displayed the second information that the user 200 has input through the virtual keyboard 400. The user 200 can continue to input information through the virtual keyboard 400. At this time, the electronic device 100 can capture the image of the user 200's hand through the camera 160, specifically, the first image can be captured through the optical camera 161, and the second image can be captured through the event camera 162. Then, the electronic device 100 can identify the key point information of the user 200's hand in combination with the first image and the second image, and obtain the second information currently input by the user 200 in combination with the second image, the key point information and the second information. The electronic device 100 can display the first information on the virtual interface 300. Among them, the key point information is, for example, the coordinate information of 21 or 22 key points of the hand, and the coordinate information can be based on the right-hand coordinate system established with the center of the user 200's head as the origin.

[0077] Based on Figure 2 The application scenario shown in the figure, the specific process of the electronic device 100 recognizing the hand movement of the user to obtain the text information input by the user is described below. Figure 3 .like Figure 3 As shown, the process may include but is not limited to the following steps:

[0078] Step 1: The electronic device processes the first image 410 and the second image 420 using a first algorithm to obtain a third image 430 .

[0079] The first image 410 is obtained by taking a picture with an optical camera. Figure 3 The first image 410 is relatively blurry and has poor image quality. The second image 420 is captured by an event camera. Figure 3 The second image 420 shown includes the outline of the moving part (ie, the user's hand).

[0080] The time when the optical camera outputs the first image 410 is the same as the time when the event camera outputs the second image 420. In some embodiments, before executing step 1, the electronic device may first select the first image 410 at the same time as the time when the event camera outputs the second image 420 from the multiple images output by the optical camera.

[0081] Specifically, the electronic device may first set the weights of the first image 410 and the second image 420, and then perform weighted summation to obtain the third image 430, that is, use the first algorithm to fuse the first image 410 and the second image 420 to obtain the third image 430, wherein the sum of the weights of the first image 410 and the second image 420 is 1. The third image 430 is used for subsequent hand key point detection, thereby realizing hand motion recognition. If the imaging effect of the first image 410 is poor (for example, low clarity, low brightness, high brightness, etc.), the electronic device may reduce the weight of the first image 410 and increase the weight of the second image 420, thereby reducing the impact of the quality of the first image 410 on the accuracy of the detection algorithm (for example, the second algorithm described in step 2, the third algorithm described in step 3) in the case of overexposure, dim light, motion blur, etc., and the result of hand motion recognition is more accurate.

[0082] In some embodiments, the electronic device may determine the weights of the first image 410 and the second image 420 according to the grayscale histogram of the first image 410. For example, it is assumed that the weights of the first image 410 and the second image 420 are both 0.5 by default. When the distribution of the grayscale histogram of the first image 410 is uneven, for example, when the distribution of the grayscale histogram is concentrated in a certain fixed interval, the clarity of the details of the first image 410 is low, and the electronic device may set the weight of the first image 410 to 0.3 and the weight of the second image 420 to 0.7.

[0083] Without limitation thereto, in a specific implementation, the electronic device may also determine the weights of the first image 410 and the second image 420 according to the average value of the first image 410. For example, when the average value of the first image 410 is greater than the first threshold value, the first image 410 is too bright, or when the average value of the first image 410 is less than the second threshold value, the first image 410 is too dark, and the electronic device may set the weight of the first image 410 to 0.4 and the weight of the second image 420 to 0.6. The electronic device may also determine the weights of the first image 410 and the second image 420 according to the standard deviation of the first image 410. For example, when the standard deviation of the first image 410 is less than the third threshold value, the contrast of the first image 410 is low, and the electronic device may set the weight of the first image 410 to 0.35 and the weight of the second image 420 to 0.65. The present application does not limit the specific method of determining the weights of the first image 410 and the second image 420.

[0084] Without being limited to the above-mentioned situations, in a specific implementation, when the electronic device uses the first algorithm to fuse the first image and the second image to obtain the third image, the weighted summation method may not be used. The present application does not limit the specific implementation method of the first algorithm.

[0085] Step 2: The electronic device identifies the target area 440 in the third image 430 using a second algorithm.

[0086] Specifically, the target area 440 is a rectangular area where the user's hand is located in the third image 430. The electronic device can use the widest line segment of the user's hand in the third image 430 as a set of opposite sides of the target area 440, and the highest line segment as another set of opposite sides of the target area 440.

[0087] Step 3: The electronic device uses a third algorithm to identify key points in the target area 440 to obtain a target area 450 including the key points.

[0088] The number and position of key points are not limited. Figure 3 In the target area 450 including key points shown, the number of key points is 21, and most of the key points are located at the joints of the user's hands.

[0089] Step 4: The electronic device uses a fourth algorithm to process the target area 450 including the key points, the second image 420 and the second information input by the user to obtain the first information currently input by the user.

[0090] It can be understood that the user can input information based on the virtual keyboard 400 presented by the electronic device. For an example where the key points of the user's hand fall on the virtual keyboard 400, see Figure 4 .

[0091] Specifically, the electronic device may first obtain the first finger movement information based on the coordinate information of the hand key points in the target area 450 including the key points. For example, the electronic device may first unify the virtual keyboard 400 and the user's hand key points into a coordinate system (for example, a right-handed coordinate system with the center of the user's left eye as the origin, for a specific example, see the following Figure 5 ). Then, the electronic device can obtain the coordinate difference (such as Euler distance, etc.) between the key points of the hand corresponding to the preset number of frames of the third image 430 and the keys on the virtual keyboard 400, and obtain the first motion information according to the coordinate difference, such as the finger that the user may perform the tapping action, the amplitude of the tapping action, the frequency of the tapping action, etc. Among them, the finger that the user may perform the tapping action generally corresponds to a key on the virtual keyboard 400, that is, the user may perform the tapping action based on the key, so when the electronic device usually obtains the finger that the user may perform the tapping action, it can obtain the word information that the user may input.

[0092] Among them, the electronic device obtains the first motion information based on the coordinate difference, for example: when a preset number of coordinate differences are all greater than the first preset difference, it is determined that the finger to which the corresponding key point belongs is the finger that may perform the tapping action. Or, among the preset number of coordinate differences, when the number of coordinate differences greater than the first preset difference is greater than the fourth threshold, it is determined that the finger to which the corresponding key point belongs is the finger that may perform the tapping action. Or, assuming that the finger that may perform the tapping action has been determined, when the corresponding coordinate difference is greater than the second preset difference, it is determined that the tapping action of the finger is more intense. Among them, the second preset difference is greater than the first preset difference. Not limited to this, when the coordinate difference is greater than the third preset difference, the fourth preset difference, and the fifth preset difference, the intensity of the tapping action can be not intense, relatively intense, and very intense, respectively. The present application does not limit the specific method of obtaining the first motion information based on the coordinate difference.

[0093] An example of an electronic device obtaining the first movement information of a finger can be seen below: Figure 5 . Figure 5 A schematic diagram showing a three-dimensional coordinate system is shown as an example. The coordinate system can be Figure 2 In the scene shown, a right-handed coordinate system is established with the center of the left eye of the user 200 as the origin. The coordinate system shows the coordinates of a key point A in two adjacent frames of the first image 410, and the output times of the two frames of the first image 410 can be the first time and the second time respectively, and the first time is earlier than the second time.

[0094] like Figure 5 As shown, the coordinate of each key on the virtual keyboard on the z-axis is z 0 , the coordinates on the x-axis and y-axis are different. The coordinate of key W (center point) on the x-axis is x 0 , the coordinate on the y-axis is y 0 The coordinates of the key point A on the x-axis at the first and second moments are both x 1 The coordinate of the key point A on the y-axis at the first moment is y 1 , the coordinate on the z-axis is z 1 The coordinate of the key point A on the y-axis at the second moment is y 2 , the coordinate z on the z-axis 2 Assume that y 2 -y 1 >y t , z 2 -z 1 >z t , where y t 、z t For the preset difference, the electronic device can determine that the finger to which the key point A belongs is the finger that may perform the tapping action.

[0095] Not limited to the above-mentioned cases, in a specific implementation, the coordinates of the key points may also be two-dimensional coordinates.

[0096] Not limited to the above-mentioned situations, in a specific implementation, the electronic device can also regard the finger to which the key point in a frame of the third image 430, which has the smallest coordinate difference with the key on the virtual keyboard 400, belongs as the finger that may perform the tapping action, so that hand motion recognition can be achieved even when the number of captured images is insufficient (for example, the storage capacity of the electronic device is small), while also reducing processing delays and providing a better user experience. Alternatively, the electronic device can also obtain the finger that may perform the tapping action based on the difference in coordinates of the same key point at different times in multiple frames of the third image 430, for example, when the coordinate difference is greater than a preset difference, the finger to which the key point belongs is the finger that may perform the tapping action. This application does not limit the specific method for determining motion information.

[0097] Understandably, Figure 3 The image used to obtain the first information currently input by the user can be a single-frame image or a multi-frame image. That is, the first image 410 and the second image 420 input into the first algorithm in step 1 and the third image 430 input into the second algorithm in step 2 can be a single-frame image at a certain moment or a multi-frame image within a certain time period.

[0098] Then, the electronic device can filter the above-mentioned first motion information according to the grayscale value of each finger position in the second image 420 to obtain the second motion information of the finger, wherein the larger the grayscale value of the finger position in the second image 420, the more violent the movement of the finger (the larger the movement amplitude and / or movement frequency). For example, in the first motion information, the user may perform the tapping action with three fingers, and the electronic device can use the two fingers with higher grayscale values ​​of the three fingers in the second image 420 as the fingers that the user may perform the tapping action in the second motion information. The electronic device can obtain the information that the user may input based on the second motion information.

[0099] For example, it is assumed that the electronic device obtains that the Euler distance between the user's index finger, middle finger and ring finger and key D, key W and key A on the virtual keyboard is the smallest, that is, in the determined first motion information, the fingers that may perform the tapping action are the index finger, middle finger and ring finger. Figure 3 The grayscale values ​​of the positions of the middle finger and the ring finger in the second image 420 are relatively high, so in the second motion information, the fingers of the user performing the tapping action are the middle finger and the ring finger. At this time, the electronic device can obtain the information that the user may input as "w" and "a" according to the second motion information.

[0100] Not limited to the above-mentioned cases, in a specific implementation, the electronic device may also first determine the second motion information of the finger according to the grayscale value of each finger position in the second image 420, for example, a finger whose grayscale value is greater than the first grayscale threshold is a finger that may perform a tapping action. Then, the electronic device screens the second motion information according to the key point information to obtain the first motion information. For example, the electronic device may calculate the coordinate difference between the key point on the finger that may perform a tapping action in the second motion information and the key on the virtual keyboard 400, and when the coordinate difference is greater than the sixth preset difference, it is determined that the finger is a finger that may perform a tapping action in the first motion information. The electronic device can obtain information that the user may input based on the first motion information.

[0101] Finally, the electronic device can obtain the first information currently input by the user based on the information that the user may input and the second information that the user has input. For example, the electronic device can use the information that the user may input and the second information that the user has input as inputs to the text judgment neural network to obtain the output first information. The first information can be information with a higher probability of input among the information that the user may input.

[0102] For example, assuming that the information that the user may input is "w" and "a", and the second information that the user has input is "abnorm", the text judgment neural network can obtain that the probability that the user currently inputs "a" is 0.8, and the probability that the user currently inputs "w" is 0.2, so the output first information is "a".

[0103] Not limited to the above-mentioned situations, in a specific implementation, if the user does not enter the second information before entering the first information, that is, the second information is empty, the electronic device can directly use the information that the user may enter as the determined first information, or judge and process the information that the user may enter to obtain the first information. This application does not limit this.

[0104] Exemplarily, the first algorithm is a proportion adjustment algorithm and an image processing algorithm, the second algorithm is a target detection neural network, the third algorithm is a key point detection neural network, and the fourth algorithm is a text judgment neural network. Not limited to this, it can also be a deep learning algorithm, etc. This application does not limit the specific forms of the first algorithm, the second algorithm, the third algorithm, and the fourth algorithm.

[0105] The above embodiments are described by taking only one hand of the user as an example. In a specific implementation, the electronic device may recognize both hands of the user at the same time to obtain information input by the user.

[0106] Not limited to the above example, in a specific implementation, the image acquired by the electronic device may not be an image of the user's hand, but an image of other parts of the user such as the legs, waist, etc. The electronic device can perform human posture recognition based on the acquired image according to the above process, improve the detection accuracy of key points, and make the recognition result more accurate.

[0107] See also Figure 6 , Figure 6 It is a flowchart of an identification method provided in an embodiment of the present application. Figure 6 Can be applied to electronic devices, the electronic devices can include an optical camera and an event camera, the electronic devices can be Figure 1 The electronic device shown. The method includes but is not limited to the following steps:

[0108] S101: The electronic device acquires a first image and a second image.

[0109] Specifically, the first image is an image captured by an optical camera, and the second image is an image captured by an event camera. The moment when the optical camera outputs the first image can be the same as the moment when the event camera outputs the second image, so the object in the first image and the second image can both be used to capture N parts of the target object moving at the same time, such as the user's hand, and the N parts are multiple fingers of the user. N is a positive integer. Based on the description of the above event camera, it can be known that the second image is determined based on the brightness change of the pixels of the moving target object, and the second image may include the moving parts of the N parts. Among them, the target object may be a user, and the N parts may be body parts of the user, such as hands, waist, legs, etc. Not limited to this, the target object may also be other organisms or objects.

[0110] S102: The electronic device obtains key point information of N parts of the target object according to the first image.

[0111] Specifically, the electronic device can identify the target area where N parts of the target object in the first image are located, and then identify key point information from the target area, wherein one set of opposite sides of the target area is the widest line segment of the N parts, and another set of opposite sides of the target area is the highest line segment of the N parts.

[0112] Without limitation to this, in a specific implementation, the electronic device may first identify the target area where P parts of the N parts are located, and then identify key point information from the target area, wherein a set of opposite sides of the target area is the widest line segment of the P parts, and another set of opposite sides of the target area is the highest line segment of the P parts. P is a positive integer, and P is less than N. The electronic device may, but is not limited to, determine the P parts based on the area occupied by the N parts in the image, clarity, etc. For example, when the area occupied by a certain part in the image is greater than a preset area and the clarity (grayscale difference or gradient of adjacent pixels) is less than a preset clarity value, the part is determined to belong to the P parts.

[0113] Specifically, the electronic device can determine the position and number of key points according to the shape of the part in the target area. Figure 3 In the target region 450 including key points shown, the number of key points on the hand (one) is 21, and most of the key points are located at the joints of the hand. Alternatively, when the part in the target region is the leg (one), the number of key points can be 10, and most of them can be located at the joints of the leg.

[0114] In some embodiments, S102 specifically includes: the electronic device obtains key point information of N parts of the target object according to the first image and the second image.

[0115] Specifically, the electronic device may first fuse the first image and the second image to obtain a third image, and then obtain key point information of N parts of the target object based on the third image. In some embodiments, the electronic device fuses the first image and the second image to obtain the third image, which may be specifically as follows: the electronic device performs weighted summation on the first image and the second image to obtain the third image. The weight of the first image may be a first weight, and the weight of the second image may be a second weight. The sum of the first weight and the second weight is 1. The electronic device may determine the first weight based on the parameters of the first image, and the parameters of the first image may include, for example but not limited to, the distribution of the grayscale histogram of the first image, the mean of the first image, and the standard deviation of the first image.

[0116] When the imaging effect of the first image is poor, the electronic device can lower the first weight and increase the second weight. At this time, the first weight can be less than the second weight. The poor imaging effect of the first image can be, but is not limited to, satisfying at least one of the following: the distribution of the grayscale histogram of the first image is concentrated in a fixed interval, the mean of the first image is greater than the first threshold, the mean of the first image is less than the second threshold, and the standard deviation of the first image is less than the third threshold. Among them, the first threshold is greater than the second threshold.

[0117] S103: The electronic device determines a first moving part among the N parts according to the key point information and the grayscale value of the second image.

[0118] Specifically, the key point information can be used to obtain information such as coordinate differences of N parts at different times, so as to obtain motion information of N parts (such as the part that may perform an action, the amplitude of the action, the frequency of the action, etc.). Based on the description of the above event camera, it can be known that the more violent the movement (the greater the amplitude and / or frequency of the movement) of the part, the greater the grayscale value of the pixel in the second image. The electronic device can obtain the first information input by the target object through the first part based on the first part. The first part can be at least one part, such as at least one finger.

[0119] In some embodiments, the electronic device may first obtain the first motion information of the target object based on the key point information, assuming that the target object is M parts of the N parts that move, where M is a positive integer and N is greater than M. Then, the electronic device may determine the first part from the M parts represented by the first motion information based on the grayscale value of the second image. The grayscale value of the pixel of the first part in the second image may be greater than the first grayscale threshold, or the grayscale value of the pixel of the first part in the second image may be greater than the grayscale value of the pixel of the other parts of the M parts.

[0120] In some embodiments, the electronic device may also first obtain the second motion information of the target object based on the grayscale value of the second image, assuming that there are T parts moving among the N parts, T is a positive integer, and N is greater than T. Among them, the grayscale values ​​of the pixels of the T parts in the second image may be greater than the second grayscale threshold, or the grayscale values ​​of the pixels of the T parts in the second image are greater than the grayscale values ​​of the pixels of other parts among the N parts. Then, the electronic device may determine the first part from the T parts represented by the second motion information based on the key point information. Optionally, the coordinate difference of the key point on the first part is greater than the first difference, or the coordinate difference of the key point on the first part is greater than the coordinate difference of the key point on other parts among the T parts.

[0121] The key point information includes, for example, the coordinates of the key point relative to a fixed point on the user or electronic device (which may be called absolute coordinates), and the coordinates of the key point relative to fixed points on N parts (which may be called relative coordinates).

[0122] In some embodiments, before S103, the method may further include: receiving second information input by a user. Then the electronic device may determine the first information based on the first part and the second information input by the user.

[0123] Figure 6 An example of the process shown can be found above. Figure 3 The process shown, where Figure 3 The following description will be given by taking the target object as a user and the N parts as the user's left hand (five fingers of the left hand) as an example.

[0124] exist Figure 6In the method shown, the electronic device can combine an optical camera and an event camera to realize key point recognition, and introduce an event camera to further determine motion information, so as to obtain the information input by the user based on the motion information, reducing the impact of the image quality of the first image on the key point detection accuracy in cases such as motion blur and similar color and texture of the object and background, and the result of motion recognition (i.e., the first information input by the user mentioned above) is also more accurate and robust. Users can input information to the electronic device and interact with the electronic device without external devices such as keyboards, wearable devices, and cameras with third-party perspectives, which enhances the interactive ability of the electronic device and makes it more convenient for users to use.

[0125] It can be understood that the embodiments of the present application do not require gesture matching and other processes, and can directly obtain the motion information of the target object based on the key point information and the image output by the event camera, and obtain the user's current input information based on the motion information and the information already input by the user, thereby avoiding misidentification, recognition failure, and other situations such as matching failure, and improving input efficiency and usability.

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above computer program product includes one or more computer instructions. When the above computer program instructions are loaded and executed on a computer, the above-mentioned process or function according to the present application is generated in whole or in part. The above computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The above computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0127] In short, the above is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

[0128] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A recognition method, characterized in that: Applied to electronic equipment, the method comprises: The electronic device acquires a first image and a second image, wherein the first image is an image captured by an optical camera, the second image is an image captured by an event camera, and the second image is determined according to brightness changes of pixels of a moving target object; The electronic device acquires key point information of N parts of the target object according to the first image, where N is a positive integer; The electronic device determines a first moving part among the N parts based on the key point information and the grayscale value of the second image, and a part with a greater movement frequency has a greater grayscale value of a pixel in the second image, wherein the grayscale value of the pixel of the first part in the second image is greater than a preset grayscale threshold, or the grayscale value of the pixel of the first part in the second image is greater than the grayscale value of the pixel of the second part in the second image, and the second part is a part among the N parts that is different from the first part.

2. The method according to claim 1, characterized in that The electronic device acquires key point information of N parts of the target object according to the first image, including: The electronic device fuses the first image and the second image to obtain a third image; The electronic device obtains key point information of the N parts according to the third image.

3. The method according to claim 2, characterized in that Before the electronic device fuses the first image and the second image to obtain a third image, the method further includes: The electronic device determines a first weight of the first image and a second weight of the second image according to parameters of the first image; the parameters of the first image include at least one of the following: distribution, mean, and standard deviation of a grayscale histogram; The electronic device fuses the first image and the second image to obtain a third image, including: The electronic device fuses the first image and the second image based on the first weight and the second weight to obtain the third image.

4. The method according to claim 3, characterized in that The electronic device determines a first weight of the first image and a second weight of the second image according to a parameter of the first image, including: When a preset condition is met, the electronic device sets the first weight and the second weight to a first preset value and a second preset value respectively, wherein the first preset value is less than the second preset value; the preset condition includes at least one of the following: the distribution of the grayscale histogram of the first image is concentrated in a fixed interval, the mean of the first image is greater than a first threshold, the mean of the first image is less than a second threshold, and the standard deviation of the first image is less than a third threshold, wherein the first threshold is greater than the second threshold.

5. The method according to claim 2, characterized in that The electronic device acquires key point information of the N parts according to the third image, including: The electronic device identifies the target area where the N parts are located in the third image; The electronic device identifies key point information of the N parts in the target area.

6. The method according to any one of claims 1 to 5, characterized in that: The electronic device determines the first moving part among the N parts according to the key point information and the grayscale value of the second image, including: The electronic device determines M parts from the N parts according to the key point information, where M is less than or equal to N and is a positive integer; the key point information includes the coordinates of at least one key point on the N parts; The electronic device determines the first part from the M parts based on the grayscale value of the second image, the grayscale value of the pixel of the first part in the second image is greater than a preset grayscale threshold, or the grayscale value of the pixel of the first part in the second image is greater than the grayscale value of the pixel of other parts of the M parts in the second image.

7. The method according to claim 6, characterized in that The difference between the coordinates of the key points on the M parts at the first moment and the coordinates at the second moment is greater than the first preset difference, and the first moment is different from the second moment; or, the difference between the coordinates of the key points on the M parts and the preset coordinates is less than the second preset difference.

8. The method according to any one of claims 1 to 5, characterized in that: The first part is used by the electronic device to determine first information input by the target object through the first part.

9. The method according to claim 8, characterized in that The method further comprises: The electronic device determines Q pieces of information according to the first part, where Q is a positive integer, and the Q pieces of information include the first information; The electronic device determines the first information from the Q pieces of information based on second information, where the second information is information input by the target object before the target object inputs the first information.

10. The method according to any one of claims 1 to 5, characterized in that: The first part is a plurality of parts.

11. The method according to any one of claims 1 to 5, characterized in that: The target object is a user, and the N parts are fingers of the user.

12. An electronic device, characterized in that: The electronic device includes one or more memories and one or more processors; the one or more memories are used to store computer programs, and the one or more processors are used to call the computer programs, and the computer program includes instructions, and when the instructions are executed by the one or more processors, the electronic device executes the method described in any one of claims 1 to 11.

13. The electronic device according to claim 12, characterized in that: The electronic device includes the event camera and the optical camera.

14. A computer storage medium, characterized in that: The method comprises a computer program, wherein the computer program comprises instructions, and when the instructions are executed on a processor, the method according to any one of claims 1 to 11 is implemented.

Images