Air gesture recognition methods and electronic devices

Abstract

This application relates to the field of image processing technology and provides a method and electronic device for air gesture recognition. The air gesture recognition method includes: acquiring information of a hand image captured by a front-facing camera at a first moment; acquiring the screen content display direction of the electronic device at the first moment; determining a target preprocessing operation corresponding to the screen content display direction; performing the target preprocessing operation on the hand in the hand image to obtain an image to be recognized; and recognizing air gestures based on multiple frames of images to be recognized. This enables the finger orientation recognized by the electronic device during air gesture recognition to be consistent with the finger orientation of the user in the actual physical space, thereby ensuring compatibility with air gesture recognition under various screen content display directions and reducing the probability of misrecognition of air gestures.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and electronic device for air gesture recognition. Background Technology

[0002] Currently, mobile phones and other electronic devices support contactless gesture interaction technology. This technology enables contactless human-computer interaction in scenarios where users cannot easily touch the input devices of the electronic device (such as touchscreens, keyboards, buttons, mice, or writing tablets), improving the convenience of operating electronic devices.

[0003] The air gesture recognition method in the relevant technology is only applicable to scenarios where the screen content of the electronic device is displayed in the direction of the screen content starting from the side where the front camera is located and along the long side of the electronic device. It is not applicable to scenarios where the screen content is displayed in other directions. In scenarios where the screen content is displayed in other directions, the probability of misrecognition of air gestures is relatively high. Summary of the Invention

[0004] This application provides a method and electronic device for air gesture recognition, which is compatible with air gesture recognition under various screen content display orientations and reduces the probability of misrecognition of air gestures.

[0005] In a first aspect, embodiments of this application provide a method for recognizing air gestures, applied to an electronic device including a front-facing camera. The air gesture recognition method includes: acquiring information of a hand image captured by the front-facing camera at a first moment; acquiring the screen content display direction of the electronic device at the first moment; determining a target preprocessing operation corresponding to the screen content display direction; performing the target preprocessing operation on the hand in the hand image to obtain an image to be recognized; and recognizing air gestures based on multiple frames of images to be recognized.

[0006] The first moment can be any moment.

[0007] A hand image can refer to an image of a human hand captured by an AO camera.

[0008] Information about a human hand image can include the image itself and the coordinates of key points on the hand.

[0009] According to the air gesture recognition method provided in the embodiments of this application, by acquiring the information of the hand image captured by the front-facing camera at the first moment and the screen content display direction of the electronic device at the first moment, and determining the target preprocessing operation corresponding to the screen content display direction, the target preprocessing operation is performed on the hand in the hand image to obtain the image to be recognized. Air gestures are recognized based on the image to be recognized. In this way, the finger orientation recognized by the electronic device during the air gesture recognition process is consistent with the finger orientation of the user in the actual physical space, thereby being compatible with air gesture recognition under various screen content display directions and reducing the probability of misrecognition of air gestures.

[0010] In one optional implementation of the first aspect, the target preprocessing operation includes a rotation operation and a left-right flip operation; determining the target preprocessing operation corresponding to the screen content display direction includes: determining the target rotation angle corresponding to the screen content display direction; performing the target preprocessing operation on the hand in the hand image includes: rotating the hand in the hand image by the target rotation angle, and then performing a left-right flip operation on the hand.

[0011] Among them, the left and right flip operation refers to the left and right mirror operation.

[0012] According to the air gesture recognition method provided in the embodiments of this application, by determining the target rotation angle corresponding to the display direction of the screen content, and rotating the hand in the hand image by the target rotation angle, the hand is flipped left and right. This can make the finger orientation of the hand in the hand image consistent with the finger orientation of the user in the actual physical space, so that the electronic device can accurately recognize the air gesture based on the finger orientation.

[0013] In one optional implementation of the first aspect, rotating the hand in the hand image by a target rotation angle and then performing a left-right flip operation on the hand includes: rotating the hand image by a target rotation angle and then performing a left-right flip operation on the hand image.

[0014] In one optional implementation of the first aspect, rotating the hand image by a target rotation angle and then performing a left-right flip operation on the hand image includes: when the resolution of the hand image is less than or equal to a first resolution and the aspect ratio of the hand image is 1:1, rotating the hand image by a target rotation angle and then performing a left-right flip operation on the hand image.

[0015] In one optional implementation of the first aspect, the information of the hand image includes the coordinates of key points of the hand corresponding to the hand image; after rotating the hand in the hand image by a target rotation angle, performing a left-right flip operation on the hand includes: determining the target coordinate transformation method corresponding to the target rotation angle and the left-right flip operation; and transforming the coordinates of the key points of the hand based on the target transformation method.

[0016] In one optional implementation of the first aspect, the information of the hand image includes the coordinates of key points of the hand corresponding to the hand image; after rotating the hand in the hand image by a target rotation angle, performing a left-right flip operation on the hand includes: if the resolution of the hand image is greater than a first resolution, or if the aspect ratio of the hand image is not 1:1, determining the target coordinate transformation method corresponding to the target rotation angle and the left-right flip operation; and transforming the coordinates of the key points of the hand based on the target transformation method.

[0017] According to the air gesture recognition method provided in the embodiments of this application, rotation and left / right flipping operations are only performed on the entire hand image when the resolution of the hand image is less than or equal to a first resolution and the aspect ratio of the hand image is 1:1. When the resolution of the hand image is greater than the first resolution, or the aspect ratio of the hand image is not 1:1, the rotation and flipping of the hand are achieved by performing coordinate transformation on the hand in the hand image. This not only reduces the computational load of electronic devices and improves the efficiency of air gesture recognition, but also saves the power consumption of electronic devices and improves the accuracy of air gesture recognition.

[0018] In one optional implementation of the first aspect, determining the target rotation angle corresponding to the screen content display direction includes: determining the target rotation angle as -90 degrees when the screen content display direction is a first display direction; determining the target rotation angle as 0 degrees when the screen content display direction is a second display direction; and determining the target rotation angle as 180 degrees when the screen content display direction is a third display direction. The first display direction indicates that the screen content is displayed along the long side of the electronic device, starting from the side where the front-facing camera is located; the second display direction indicates that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the left of the screen content display direction; and the third display direction indicates that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the right of the screen content display direction. A negative target rotation angle indicates counter-clockwise rotation, and a positive target rotation angle indicates clockwise rotation.

[0019] In one optional implementation of the first aspect, determining the target coordinate transformation method corresponding to the target rotation angle and the left / right flip operation includes: when the target rotation angle is -90 degrees, determining the target coordinate transformation method as: x = h-1 - y_raw, y = w-1 - x_raw; when the target rotation angle is 0 degrees, determining the target coordinate transformation method as: x = w-1 - x_raw, y = y_raw; when the target rotation angle is -180 degrees, determining the target coordinate transformation method as: x = w-1 - x_raw, y = h-1 - y_raw; where x_raw represents the abscissa of the hand keypoint before the coordinate transformation, y_raw represents the ordinate of the hand keypoint before the coordinate transformation, h represents the height of the hand image, w represents the width of the hand image, x represents the abscissa of the hand keypoint after the coordinate transformation, and y represents the ordinate of the hand keypoint after the coordinate transformation.

[0020] Secondly, embodiments of this application provide an electronic device, including: one or more processors, and a memory;

[0021] The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors calling the computer instructions to cause the electronic device to perform the air gesture recognition method as described in any implementation of the first aspect above.

[0022] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the air gesture recognition method as described in any implementation of the first aspect above.

[0023] Fourthly, embodiments of this application provide a computer-executable program product that, when running on an electronic device, causes the electronic device to execute the air gesture recognition method of any implementation of the first aspect described above.

[0024] Fifthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the air gesture recognition method as described in any implementation of the first aspect above.

[0025] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0026] Figure 1 A schematic diagram illustrating the placement posture of an electronic device according to an embodiment of this application;

[0027] Figure 2 This is a schematic diagram illustrating the screen content display direction of an electronic device in different placement postures when the automatic screen rotation function is enabled, as provided in an embodiment of this application.

[0028] Figure 3 This is a schematic diagram illustrating the screen content display direction of an electronic device in different placement postures when the automatic screen rotation function is not enabled, as provided in an embodiment of this application.

[0029] Figure 4 A schematic diagram illustrating a scenario of controlling an electronic device via air gestures, provided as an embodiment of this application;

[0030] Figure 5 A schematic diagram illustrating a scenario where a user inputs air gestures under different screen content display orientations, as provided in an embodiment of this application.

[0031] Figure 6 A schematic diagram illustrating a scenario where a user inputs air gestures under another screen content display orientation provided in this application embodiment;

[0032] Figure 7 A schematic diagram illustrating a scenario where a user inputs air gestures under different screen content display orientations, as provided in an embodiment of this application.

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0034] Figure 9 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0035] Figure 10 A schematic flowchart illustrating a gesture recognition method provided in an embodiment of this application;

[0036] Figure 11 This is a schematic diagram illustrating the interaction timing between various modules in the system architecture of an electronic device during the implementation of a gesture recognition method provided in this application embodiment.

[0037] Figure 12 This is a schematic diagram illustrating the interaction timing between modules in the system architecture of an electronic device during the implementation of a gesture recognition method provided in another embodiment of this application. Detailed Implementation

[0038] It should be noted that the terminology used in the implementation section of the embodiments of this application is only used to explain the specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related items, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more.

[0039] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] To facilitate understanding, the relevant concepts involved in the embodiments of this application will be explained first.

[0042] 1. The placement posture of electronic devices

[0043] The way an electronic device is positioned can be used to indicate how a user handles the device. For example, using a mobile phone as an example, please refer to [link to relevant documentation]. Figure 1 This is a schematic diagram illustrating the placement posture of an electronic device according to an embodiment of this application. The placement posture of the electronic device may include, for example, vertical placement with the front-facing camera at the top, such as... Figure 1 (a) in the example; horizontal placement with the front-facing camera on the left, such as... Figure 1 (b) in the example; horizontal placement with the front camera on the right, such as... Figure 1 (c) in the middle.

[0044] 2. Automatic screen rotation function

[0045] Electronic devices can support automatic screen rotation. When automatic screen rotation is enabled, the electronic device automatically rotates the screen to adapt to its orientation, ensuring the screen content is aligned with the user's facial orientation. For an example, please refer to [link to example]. Figure 2 This is a schematic diagram illustrating the screen content display direction of an electronic device in different placement postures when the automatic screen rotation function is enabled, as provided in an embodiment of this application.

[0046] With the screen auto-rotation function enabled, such as Figure 2 As shown in (a), if the electronic device is placed vertically with the front-facing camera at the top, the screen content can be displayed along the long side of the electronic device, starting from the side where the front-facing camera is located. Figure 2 As shown in (b), if the electronic device is placed horizontally and the front-facing camera is located to the left of the screen content display direction, then the screen content display direction can be: displayed along the short side of the electronic device with the front-facing camera located to the left of the screen content display direction. Figure 2 As shown in (c), if the electronic device is placed horizontally and the front-facing camera is located on the right side of the screen content display direction, then the screen content display direction can be: displayed along the short side of the electronic device and the front-facing camera is located on the right side of the screen content display direction.

[0047] For example, please refer to Figure 3 This is a schematic diagram illustrating the screen content display orientation of an electronic device in different placement postures when the screen auto-rotation function is not enabled, as provided in an embodiment of this application. When the screen auto-rotation function is not enabled, such as... Figure 3 As shown in (a), (b) and (c), regardless of the placement of the electronic device, the screen content is displayed in the following direction: starting from the side where the front-facing camera is located, and displaying along the long side of the electronic device.

[0048] Therefore, the way electronic devices display their screen content can be determined by the orientation of the electronic device and whether the screen rotation function is enabled.

[0049] Currently, mobile phones and other electronic devices support contactless gesture interaction technology. This technology enables contactless human-computer interaction in scenarios where users cannot easily touch the input devices of the electronic device (such as touchscreens, keyboards, buttons, mice, or writing tablets), improving the convenience of operating electronic devices.

[0050] Specifically, electronic devices can support always-on (AO) camera functionality. When AO is enabled, the front-facing camera of the electronic device is always on and can capture images in real time. The electronic device can analyze multiple frames of images captured by the AO camera (i.e., the always-on front-facing camera) to recognize user-inputted air gestures and can respond to the recognized air gestures by performing corresponding operations, thus enabling air control of the electronic device.

[0051] For example, please refer to Figure 4 This is a schematic diagram illustrating a scenario where an electronic device is controlled via air gestures, as provided in an embodiment of this application. Assuming that... Figure 4 The palm with fingers pointing upwards, shown in (a) in the image, becomes... Figure 4 The clenched fist gesture shown in (b) corresponds to a screenshot operation. Therefore, the electronic device can perform a screenshot operation after recognizing the clenched fist gesture, obtaining a screenshot image, thus achieving air screenshotting. Assuming that... Figure 4 The palm with fingers pointing upwards, as shown in (c), becomes... Figure 4 As shown in (d), a downward-pointing hand gesture corresponds to a swipe-down screen operation or a page-down operation. Therefore, the electronic device can execute the swipe-down screen operation or page-down operation after recognizing the swipe-down gesture, thus achieving a non-invisible swipe-down screen operation or page-down operation. Assuming that... Figure 4 The back of the hand with fingers pointing downwards, as shown in (e), becomes... Figure 4 The upward swipe gesture of the palm with the fingers pointing upward, as shown in (f), corresponds to the swipe-up screen operation or the page-turning operation. The electronic device can then perform the swipe-up screen operation or the page-turning operation after recognizing the upward swipe gesture, thereby realizing the screen swipe-up or page-turning operation without air contact.

[0052] As can be seen, each air gesture corresponds to a dynamic process of hand shape change. To facilitate the identification of air gestures, the multiple hand shapes involved in air gestures can be divided into starting gestures, intermediate gestures, and ending gestures. The starting gesture can refer to the initial action when the user performs the air gesture, the ending gesture can refer to the final action when the user completes the air gesture, and the intermediate gestures can include the various hand shapes experienced during the process of changing from the starting gesture to the ending gesture.

[0053] For example, in combination Figure 4 , Figure 4 The upward-pointing palm shown in (a) can be the starting hand gesture for a grasping motion. Figure 4 The fist shown in (b) can be the finishing gesture of a grasping hand gesture. Figure 4 The upward-pointing palm shown in (c) can be the starting hand gesture for the downward swipe gesture. Figure 4 The hand with the fingers pointing downwards, as shown in (d) in the image, can be the finishing gesture for the downward swipe gesture. Figure 4 The hand with fingers pointing downwards, as shown in (e) in the diagram, can be the starting hand position for the upward swipe gesture. Figure 4 The palm with fingers pointing upwards, shown in (f), can be the ending gesture for an upward swipe gesture. It is evident that at least one of the starting and ending gestures for different air gestures is different. Based on this, electronic devices can at least recognize air gestures based on the starting and ending gestures. However, to accurately recognize air gestures, it is necessary to first accurately recognize the starting and / or ending gestures.

[0054] Typically, electronic devices can determine whether a gesture in an image is a starting gesture based on the type of hand and the direction of the fingers. For example, such as... Figure 4 As shown in (a) and (c), when the hand category corresponding to the image is palm and the fingers are pointing upwards, it can be determined that the gesture in the image is the starting gesture of a grasping gesture or a sliding gesture, etc.

[0055] However, users may input air gestures even when the screen content of an electronic device is displayed in different orientations. For example, please refer to... Figure 5 Users may input air gestures when the screen content is displayed along the long side of the electronic device, starting from the side where the front-facing camera is located. For example, please refer to... Figure 6 Users may input air gestures when the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the left of the screen content. For example, please refer to... Figure 7 Users may input air gestures when the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the right of the screen content display direction.

[0056] When the screen orientation of an electronic device is different, the AO camera processes the captured image differently, resulting in a discrepancy between the finger orientation in the AO camera's output image and the actual finger orientation of the user in physical space. For example, in Figure 5 In the scenario shown, assuming the user's input gesture is a palm with fingers pointing upwards, that is, the user's fingers are facing upwards in the actual physical space. Figure 5 (e) in the image indicates upward orientation; however, the AO camera will output... Figure 5 As shown in (d) in the image, if the gesture recognition is performed directly based on the image output by the AO camera, it is easy to identify a palm with fingers pointing upwards as a palm with fingers pointing to the left, resulting in misidentification.

[0057] It is evident that the air gesture recognition method in the relevant technology is only applicable to scenarios where the screen content of the electronic device is displayed in the direction of the screen content starting from the side where the front camera is located and along the long side of the electronic device. It is not applicable to scenarios where the screen content is displayed in other directions. In scenarios where the screen content is displayed in other directions, the probability of misrecognition of air gestures is relatively high.

[0058] In view of this, embodiments of this application provide a method and electronic device for air gesture recognition. By acquiring information of a hand image captured by a front-facing camera at a first moment and the screen content display direction of the electronic device at the first moment, and determining the target preprocessing operation corresponding to the screen content display direction, the target preprocessing operation is performed on the hand in the hand image to obtain an image to be recognized. Air gestures are recognized based on the image to be recognized. This enables the finger orientation recognized by the electronic device during the air gesture recognition process to be consistent with the finger orientation of the user in the actual physical space, thereby being compatible with air gesture recognition under various screen content display directions and reducing the probability of misrecognition of air gestures.

[0059] The air gesture recognition method provided in this application embodiment can be applied to electronic devices including front-facing cameras. Electronic devices may include mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs), etc. This application embodiment does not limit the specific type of electronic device.

[0060] For example, please refer to Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0061] like Figure 8As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0062] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0063] For example, processor 110 can be used to execute the air gesture recognition method in the embodiments of this application.

[0064] A controller can be the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0065] The processor 110 may also include a memory for storing instructions and data. For example, the memory may include a first storage area and a second storage area. The first storage area may be used to store the frame information recording queue involved in the air gesture recognition process; the second storage area may be used to store the recognition time and type of each recognized valid air gesture.

[0066] Camera 193 can be used to capture still images or videos. An electronic device may include one or N cameras 193, where N is a positive integer greater than 1. At least one of the N cameras 193 is a front-facing camera.

[0067] Display screen 194 is used to display images, videos, hand icons involved in air gesture recognition, etc. Display screen 194 may include a display panel. Electronic devices can implement display functions through GPU, display screen 194, and application processor, etc.

[0068] It is understood that the above is an exemplary description of the structure of an electronic device. It should be understood that in other embodiments, the electronic device may include more or fewer components than illustrated, or may combine or separate certain components, or may have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0069] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software architecture of the electronic device.

[0070] Please see Figure 9 This is a schematic diagram of the software architecture of an electronic device provided in an embodiment of this application.

[0071] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system can be divided into four layers, from top to bottom: the application layer, the application framework layer, the system runtime library layer, and the kernel layer.

[0072] The application layer can include a series of application packages, such as settings, camera, and smart sensing application packages. For ease of explanation, these application packages will be referred to as applications from now on.

[0073] Intelligent sensing applications can be used to support gesture control services. Gesture control services can include display services related to gesture control, such as displaying a hand icon corresponding to the initial gesture of the gesture; or, gesture control services can include response services for responding to gesture control, such as performing operations corresponding to the gesture control.

[0074] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0075] For example, the application framework layer may include a window manager, a message manager, a sensor service, etc.

[0076] A window manager can be used to manage windowed applications. For example, a window manager can be used to get the screen size, determine if there is a status bar, lock the screen, and capture the screen.

[0077] A message manager can be used to pass messages between different components. Specifically, a message manager can decouple different components through a message publish and subscribe mechanism, allowing one component to communicate directly with another component without calling the other, by publishing messages.

[0078] For example, smart sensing applications can subscribe to messages related to air gesture services through a message manager.

[0079] Sensor services can be used to manage and provide sensor data, such as data from accelerometers, magnetometers, and gyroscopes in the hardware layer, and to provide data from each sensor to other modules.

[0080] The system runtime library layer may include air gesture algorithm libraries, etc.

[0081] The air gesture algorithm library can be used to recognize air gestures based on multiple consecutive frames of images captured by an AO camera at the hardware layer. For example, the air gesture algorithm library may include a hand feature acquisition module, a preprocessing module, and an air gesture recognition module.

[0082] The hand feature acquisition module may include a hand target detection unit, a hand classification unit, and a hand key point detection unit.

[0083] It should be noted that the specific functions of the hand target detection unit, hand classification unit, hand key point detection unit, preprocessing module, and air gesture recognition module will be introduced in subsequent embodiments and will not be detailed here.

[0084] The kernel layer is the layer between hardware and software. The kernel layer can contain camera drivers, sensor drivers, etc.

[0085] It should be noted that, Figure 9 Only the modules relevant to the embodiments of this application are shown. In other embodiments, each layer may include any other possible modules, and each module may include one or more sub-modules. This application does not limit these modules.

[0086] Please see Figure 10 This is a schematic flowchart illustrating a gesture recognition method using air gestures provided in an embodiment of this application. The gesture recognition method may include steps S101 to S105, as detailed below:

[0087] S101, acquire information about the hand image captured by the AO camera at the first moment.

[0088] The first moment can be any moment.

[0089] A hand image can refer to an image of a human hand captured by an AO camera.

[0090] Information about a human hand image can include the image itself and the coordinates of key points on the hand.

[0091] S102, Obtain the screen content display orientation of the electronic device at the first moment.

[0092] Optionally, the electronic device can obtain the screen content display orientation of the electronic device at a given moment from the message manager.

[0093] For example, the screen content display direction may include a first display direction, a second display direction, and a third display direction.

[0094] Combination Figure 5 The first display orientation can be used to indicate that the screen content starts from the side where the front-facing camera is located and is displayed along the long side of the electronic device. Combined with... Figure 6 The second display orientation can be used to indicate that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the left of the screen content display orientation. Combined with... Figure 7 The third display orientation can be used to indicate that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the right of the screen content display orientation.

[0095] S103, determine the target preprocessing operation corresponding to the screen content display direction.

[0096] For example, target preprocessing operations may include rotation operations and left-right flip operations.

[0097] Based on this, S103 may specifically include: determining the target rotation angle corresponding to the screen content display direction.

[0098] Optionally, the electronic device may store the correspondence between the screen content display direction and the rotation angle. For example, the correspondence between the screen content display direction and the rotation angle may be as shown in Table 1.

[0099] Table 1

[0100]

[0101]

[0102] A negative rotation angle can be used to represent counterclockwise rotation, while a positive rotation angle can be used to represent clockwise rotation.

[0103] Based on this, electronic devices can determine the target rotation angle corresponding to the screen content display in the following way:

[0104] With the screen content display orientation set to the first display orientation, the target rotation angle is determined to be -90 degrees;

[0105] When the screen content is displayed in the second display direction, the target rotation angle is determined to be 0 degrees;

[0106] With the screen content displayed in the third display direction, the target rotation angle is determined to be -180 degrees.

[0107] S104, Perform target preprocessing operation on the hand in the hand image to obtain the image to be recognized.

[0108] For example, when the target processing operation includes a rotation operation and a left-right flip operation, S104 may specifically include:

[0109] After rotating the hand in the image by the target rotation angle, perform a left-right flip operation on the hand.

[0110] Optionally, the electronic device can rotate and flip the hand in an image by directly rotating and flipping the entire image. Based on this, in one optional implementation, after rotating the hand in the image by a target rotation angle and then performing a left-right flip operation, it can include:

[0111] After rotating the hand image by the target rotation angle, perform a left-right flip operation on the hand image.

[0112] Understandably, when the resolution of a hand image is high, directly rotating and flipping the image increases the computational load on electronic devices, thus reducing the efficiency of air gesture recognition and increasing power consumption. Furthermore, when the aspect ratio of the hand image is not 1:1, directly rotating and flipping it alters the content, reducing the accuracy of air gesture recognition. Therefore, to improve the efficiency and accuracy of air gesture recognition, after rotating the hand image by the target rotation angle, a left-right flip operation is performed. Specifically, this may include:

[0113] If the resolution of the hand image is less than or equal to the first resolution and the aspect ratio of the hand image is 1:1, rotate the hand image by the target rotation angle and then perform a left-right flip operation on the hand image.

[0114] This implementation can be performed after the AO camera outputs an image of the hand.

[0115] The first resolution can be set according to actual needs, and there is no limitation on it here.

[0116] For example, please continue reading Figure 5 With the screen content displayed in the first display orientation, assuming the user inputs a palm with fingers pointing upwards, the AO camera will output the image of the hand as follows: Figure 5 The image shown in (d) is obtained by rotating the image of the hand counterclockwise by 90 degrees and then flipping it horizontally using an electronic device. Figure 5 The image shown in (e) is... Figure 5 The finger orientation in the image shown in (e) is consistent with the finger orientation of the user in actual physical space.

[0117] For example, please continue reading Figure 6 When the screen content is displayed in the second display orientation, assuming the user inputs a palm with fingers pointing upwards, the hand image output by the AO camera will be: Figure 6 The image shown in (b) can be obtained by flipping the image of a human hand horizontally using an electronic device. Figure 6 The image shown in (c) is... Figure 6 The finger orientation in the image shown in (c) is consistent with the finger orientation of the user in actual physical space.

[0118] For example, please continue reading Figure 7 When the screen content is displayed in the third display orientation, assuming the user inputs a palm with fingers pointing upwards, the hand image output by the AO camera will be: Figure 7 The image shown in (b) is obtained by rotating the image of a human hand 180 degrees counterclockwise and then flipping it horizontally using an electronic device. Figure 7 The image shown in (c) is... Figure 7 The finger orientation in the image shown in (c) is consistent with the finger orientation of the user in actual physical space.

[0119] Optionally, if the resolution of the hand image is greater than the first resolution, or if the aspect ratio of the hand image is not 1:1, the electronic device may only rotate and flip the hand in the hand image. Based on this, in another optional implementation, after rotating the hand in the hand image by the target rotation angle, performing a left-right flip operation on the hand may include:

[0120] Determine the target coordinate transformation method corresponding to the target rotation angle and left / right flip operation;

[0121] The coordinates of key points on the hand are transformed based on the target transformation method.

[0122] This implementation method can be executed after obtaining the coordinates of the key points of the human hand in the image.

[0123] Optionally, the electronic device may store the correspondence between rotation angles and left / right flip operations and coordinate transformation methods. For example, the correspondence between rotation angles and left / right flip operations and coordinate transformation methods may be shown in Table 2.

[0124] Table 2

[0125]

[0126] Here, x_raw can be used to represent the x-coordinate of the hand keypoint in the preset coordinate system, and y_raw can be used to represent the y-coordinate of the hand keypoint in the preset coordinate system. For an example, please refer to [link to example]. Figure 8 This is a schematic diagram of a preset coordinate system provided in an embodiment of this application. The preset coordinate system can be a Cartesian coordinate system established with the top-left vertex O of the hand image 81 as the origin, and the straight lines containing the two sides intersecting at the top-left vertex O as the x-axis and y-axis, respectively. Specifically, the straight line containing the side OL, which is parallel to the horizontal plane among the two sides intersecting at the top-left vertex, can be used as the x-axis of the preset coordinate system, and the straight line containing the side OH, which is perpendicular to the horizontal plane among the two sides intersecting at the top-left vertex, can be used as the y-axis of the preset coordinate system.

[0127] h can be used to represent the height of a human hand image, and w can be used to represent the width of a human hand image.

[0128] x can be used to represent the x-coordinate of the key point of the hand after coordinate transformation, and y can be used to represent the y-coordinate of the key point of the hand after coordinate transformation.

[0129] S105 recognizes air gestures based on multiple frames of images to be recognized.

[0130] It should be noted that the specific method of recognizing air gestures based on multiple frames of images to be recognized is existing technology, and the details can be found in the relevant descriptions in the existing technology, which will not be elaborated here.

[0131] Please see Figure 11 This diagram illustrates the interaction timing between modules in the system architecture of an electronic device during the implementation of a gesture recognition method provided in this application. For example, during the implementation of the gesture recognition method, the AO camera, preprocessing module, hand feature acquisition module, and gesture recognition module can interact. The specific interaction process may include S111 to S118, detailed below:

[0132] S111, the AO camera acquires multiple consecutive frames of images and sends each frame to the preprocessing module in sequence according to the acquisition time.

[0133] Among them, the acquisition timing can be used to indicate the order in which the AO camera acquires each frame of images.

[0134] S112, the preprocessing module obtains the screen content display direction.

[0135] The preprocessing module can obtain the screen content display orientation from the message manager.

[0136] S113, the preprocessing module obtains the target rotation angle corresponding to the screen content display direction.

[0137] It should be noted that S113 is Figure 10 One specific embodiment of S103 is shown. For details regarding S113, please refer to [link / reference]. Figure 10 The relevant descriptions in S103 shown will not be repeated here.

[0138] S114, the preprocessing module rotates the image by the target rotation angle for each frame, performs a left-right flip operation on the image to obtain the image to be recognized, and sends the image to be recognized to the hand target detection unit.

[0139] It should be noted that S114 and Figure 10 In S104 shown, the steps of rotating the hand in the hand image by the target rotation angle and then performing a left-right flipping operation on the hand are similar. For details, please refer to the relevant description in S104, which will not be repeated here.

[0140] S115, the hand target detection unit performs hand target detection on the image to be recognized, obtains the information of the hand detection box corresponding to the image to be recognized, and sends the information of the hand detection box corresponding to the image to be recognized to the hand classification unit, the hand key point detection unit, and the air gesture recognition module.

[0141] It is understandable that, since the presence of a human hand in each frame of the image captured by the AO camera is uncertain, during the air gesture recognition process, after receiving each frame of the image from the AO camera, the hand feature acquisition module can first use the hand target detection unit to preliminarily identify images containing a human hand and determine the information of the hand detection box corresponding to the image containing the human hand. It should be noted that the specific content of the hand detection box information can be referred to the relevant description in the foregoing embodiments, and will not be repeated here.

[0142] In this embodiment, the purpose of the hand target detection unit outputting the hand detection box information is to enable the hand classification unit or the hand key point detection unit to locate the hand in the corresponding image based on the hand detection box information, so as to facilitate the hand classification unit, the hand key point detection unit or the air gesture recognition module to perform subsequent operations on the image.

[0143] Optionally, the hand target detection unit can perform hand target detection on the image to be recognized based on the object detection (OD) algorithm to obtain the information of the hand detection box corresponding to the image to be recognized.

[0144] S116, the hand classification unit classifies the hands in the image to be recognized based on the information of the hand detection box corresponding to the image to be recognized, obtains the hand category corresponding to the image to be recognized, and sends the hand category corresponding to the image to be recognized to the air gesture recognition module.

[0145] For example, the hand classification unit can be configured with a pre-trained hand classification model. This model can be used to classify hands to determine their category. Therefore, for each frame of an image, the hand classification unit can use the pre-trained hand classification model to determine the hand category corresponding to the image to be recognized.

[0146] In one alternative implementation, in order to reduce the computational load and improve the efficiency of air gesture recognition, for each frame of image, the hand classification unit can crop out a partial image of the hand that only includes the hand from the image to be recognized based on the information of the hand detection box corresponding to the image to be recognized, and input the partial image of the hand corresponding to the image to be recognized into the pre-trained hand classification model to obtain the hand category corresponding to the image to be recognized.

[0147] For example, the hand classification model can be a neural network module trained based on a deep learning algorithm. This application does not limit the specific type of hand classification model or the training method.

[0148] S117, the hand key point detection unit performs hand key point detection on the image to be recognized based on the information of the hand detection box corresponding to the image to be recognized, obtains the coordinates of the hand key points corresponding to the image to be recognized, and sends the coordinates of the hand key points corresponding to the image to be recognized to the air gesture recognition module.

[0149] For example, the hand keypoint detection unit can be configured with a pre-trained hand keypoint detection model. This model can detect hand keypoints in an image and output their coordinates. Therefore, for each frame, the hand keypoint detection unit can obtain the coordinates of the hand keypoints corresponding to the image to be recognized using the pre-trained model.

[0150] In one optional implementation, in order to reduce the computational load and improve the efficiency of air gesture recognition, for each frame of image, the hand key point detection unit can crop out a partial image of the hand that only includes the hand from the image to be recognized based on the information of the hand detection box corresponding to the image to be recognized, and input the partial image of the hand corresponding to the image to be recognized into the hand key point detection model to obtain the coordinates of the hand key points corresponding to the image to be recognized.

[0151] S118, the air gesture recognition module recognizes air gestures based on information from multiple frames of images to be recognized.

[0152] For example, different air gestures can be configured with different recognition strategies. The recognition strategy can be used to represent the characteristics of hand shape changes in air gestures.

[0153] Based on this, in a specific implementation, air gesture recognition can determine whether there are consecutive multi-frame images to be recognized that match the hand shape change characteristics of any air gesture, based on the recognition strategy of each air gesture.

[0154] Optionally, if there are multiple consecutive frames of images to be recognized that match the hand shape change characteristics of the target air gesture, the air gesture recognition module can determine that an air gesture currently exists and can identify the type of the target air gesture as the type of the currently recognized air gesture. For example, assuming there are multiple consecutive frames of images to be recognized that match the hand shape change characteristics of an up swipe gesture, the air gesture recognition module can determine that an up swipe gesture currently exists.

[0155] Optionally, if there are no consecutive frames of images to be recognized that match the hand shape change characteristics of any air gesture, the air gesture recognition module can determine that there is currently no air gesture.

[0156] Please see Figure 12 This is a schematic diagram illustrating the interaction timing between modules in the system architecture of an electronic device during the implementation of a gesture recognition method according to another embodiment of this application. This embodiment and... Figure 11 The difference between the corresponding embodiments is that, Figure 11 The corresponding embodiment achieves target preprocessing of the human hand in the image by performing target preprocessing operations (including rotation and left / right flipping) on ​​the entire image. This embodiment, however, achieves target preprocessing of the human hand in the image by performing coordinate transformation on the coordinates of the corresponding key points of the human hand. This saves electronic computing resources, reduces the power consumption of electronic devices, and improves the efficiency and accuracy of air gesture recognition.

[0157] Specifically, such as Figure 12 As shown, in the implementation of the air gesture recognition method, the AO camera, the hand feature acquisition module, the preprocessing module, and the air gesture recognition module can interact. The specific interaction process can include S121 to S129, detailed below:

[0158] S121, the AO camera acquires multiple consecutive frames of images and sends each frame of images to the hand target detection unit in sequence according to the acquisition time.

[0159] It should be noted that S121 and Figure 11 The corresponding embodiment of S111 is similar; please refer to [link / reference] for details. Figure 11 The relevant descriptions of part S111 in the corresponding embodiments will not be repeated here.

[0160] S122, according to the acquisition time sequence, for each frame of the image, the hand target detection unit performs hand target detection on the current frame image, obtains the information of the hand detection box corresponding to the current frame image, and sends the information of the hand detection box corresponding to the current frame image to the hand classification unit, the hand key point detection unit and the preprocessing module.

[0161] It should be noted that S122 and Figure 11 The corresponding embodiment has a similar S115, which can be found in the following examples. Figure 11 The relevant descriptions of part S115 in the corresponding embodiments will not be repeated here.

[0162] S123, according to the acquisition sequence, for each frame of image, the hand classification unit classifies the hands in the current frame image based on the information of the hand detection box corresponding to the current frame image, obtains the hand category corresponding to the current frame image, and sends the hand category corresponding to the current frame image to the preprocessing module.

[0163] It should be noted that S123 and Figure 11 S116 in the corresponding embodiment is similar; please refer to [link / reference] for details. Figure 11 The relevant descriptions of part S116 in the corresponding embodiments will not be repeated here.

[0164] S124, according to the acquisition sequence, for each frame of the image, the hand key point detection unit performs hand key point detection on the current frame image based on the information of the hand detection box corresponding to the current frame image, obtains the coordinates of the hand key points corresponding to the current frame image, and sends the coordinates of the hand key points corresponding to the current frame image to the preprocessing module.

[0165] It should be noted that S124 and Figure 11 S117 in the corresponding embodiment is similar; please refer to [link / reference] for details. Figure 11 The relevant descriptions of part S117 in the corresponding embodiments will not be repeated here.

[0166] S125, the preprocessing module obtains the display direction of the screen content.

[0167] S126, The preprocessing module obtains the target rotation angle corresponding to the screen content display direction.

[0168] S127, the preprocessing module determines the target coordinate transformation method corresponding to the target rotation angle and left and right flip operations.

[0169] S128, the preprocessing module performs coordinate transformation on the current frame image based on the target coordinate transformation method to obtain the image to be recognized, and sends the information of the image to be recognized to the air gesture recognition module.

[0170] It should be noted that S127~S128 and Figure 10 In S104 shown, the target coordinate transformation method corresponding to the target rotation angle and left and right flip operation is determined. The steps for transforming the coordinates of the key points of the human hand based on the target transformation method are similar. For details, please refer to the relevant description in S104, which will not be repeated here.

[0171] The information of the image to be recognized sent by the preprocessing module to the air gesture recognition module may include: the information of the hand detection box corresponding to the image to be recognized, the hand category, and the coordinates of the key points of the hand after coordinate transformation.

[0172] S129, the air gesture recognition module recognizes air gestures based on information from multiple frames of images to be recognized.

[0173] It should be noted that S129 and Figure 11 The corresponding embodiment has a similar S118, which can be found in the following examples. Figure 11 The relevant descriptions of part S118 in the corresponding embodiments will not be repeated here.

[0174] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, which, when invoked by a computer, causes the computer to perform one or more steps in any of the above method embodiments.

[0175] Based on the same technical concept, embodiments of this application also provide a chip system, including a processor coupled to a memory, which executes a computer-executable program stored in the memory to implement one or more steps in any of the above method embodiments. This chip system can be a single chip or a chip module composed of multiple chips.

[0176] Based on the same technical concept, this application also provides a computer executable program product that, when run on an electronic device, causes the electronic device to perform one or more steps in any of the above method embodiments.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0178] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0179] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0180] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A method for recognizing air gestures, characterized in that, The air gesture recognition method, applied to electronic devices including a front-facing camera, includes: Obtain information about the hand image captured by the front-facing camera at the first moment; With the automatic screen rotation function enabled, the screen content display orientation of the electronic device at the first moment is obtained; Determine the target preprocessing operation corresponding to the screen content display direction; If the resolution of the hand image is less than or equal to the first resolution and the aspect ratio of the hand image is 1:1, the target preprocessing operation is performed on the hand image to obtain the image to be recognized. If the resolution of the hand image is greater than the first resolution, or if the aspect ratio of the hand image is not 1:1, the target preprocessing operation is performed on the hand in the hand image to obtain the image to be identified. Based on multiple frames of the image to be identified, air gestures are recognized.

2. The air gesture recognition method according to claim 1, characterized in that, The target preprocessing operations include rotation operations and left / right flip operations; Determine the target preprocessing operation corresponding to the screen content display orientation, including: Determine the target rotation angle corresponding to the screen content display direction; The step of performing the target preprocessing operation on the hand in the hand image includes: After rotating the hand in the image by the target rotation angle, the hand is flipped left and right.

3. The air gesture recognition method according to claim 2, characterized in that, After rotating the hand in the hand image by a target rotation angle, perform a left-right flipping operation on the hand, including: After rotating the hand image by the target rotation angle, the hand image is flipped left and right.

4. The air gesture recognition method according to claim 2, characterized in that, The information in the hand image includes the coordinates of key points of the hand corresponding to the hand image; after rotating the hand in the hand image by a target rotation angle, performing a left-right flipping operation on the hand includes: Determine the target coordinate transformation method corresponding to the target rotation angle and left / right flip operation; The coordinates of the key points of the human hand are transformed based on the target coordinate transformation method.

5. The air gesture recognition method according to any one of claims 2-4, characterized in that, Determining the target rotation angle corresponding to the screen content display orientation includes: When the screen content display direction is the first display direction, the target rotation angle is determined to be -90 degrees; When the screen content display direction is the second display direction, the target rotation angle is determined to be 0 degrees; When the screen content display direction is the third display direction, the target rotation angle is determined to be 180 degrees; Wherein, the first display direction is used to indicate that the screen content starts from the side where the front-facing camera is located and is displayed along the long side of the electronic device; the second display direction is used to indicate that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the left of the screen content display direction; the third display direction is used to indicate that the screen content is displayed along the short side of the electronic device and the front-facing camera is located to the right of the screen content display direction; the target rotation angle is negative to indicate counterclockwise rotation, and the target rotation angle is positive to indicate clockwise rotation.

6. The air gesture recognition method according to claim 4, characterized in that, Determining the target coordinate transformation method corresponding to the target rotation angle and left / right flip operation includes: When the target rotation angle is -90 degrees, the target coordinate transformation method is determined to be: x=h-1-y_raw, y=w-1-x_raw; When the target rotation angle is 0 degrees, the target coordinate transformation method is determined as: x = w-1 - x_raw, y = y_raw; When the target rotation angle is -180 degrees, the target coordinate transformation method is determined to be: x=w-1-x_raw, y=h-1-y_raw; Where x_raw represents the x-coordinate of the key point of the hand before coordinate transformation, y_raw represents the y-coordinate of the key point of the hand before coordinate transformation, h represents the height of the hand image, w represents the width of the hand image, x represents the x-coordinate of the key point of the hand after coordinate transformation, and y represents the y-coordinate of the key point of the hand after coordinate transformation.

7. An electronic device, characterized in that, include: One or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

Images