Gesture key point recognition method and device, head-mounted equipment and storage medium

Abstract

The invention provides a gesture key point recognition method and device, head-mounted equipment and a computer readable storage medium. The gesture key point recognition method comprises the following steps: acquiring a first view-finding picture of the head-mounted equipment; carrying out key point detection based on the first view-finding picture to obtain an image two-dimensional coordinate of an operation key point of a target gesture in the first view-finding picture and an image two-dimensional coordinate of a reference key point of the target gesture; performing projection processing based on the image two-dimensional coordinates of the operation key points and the image two-dimensional coordinates of the reference key points to obtain second projection three-dimensional coordinates of the operation key points; obtaining the overall offset of the target gesture; and performing conversion processing according to the overall offset of the target gesture and the second projection three-dimensional coordinate of the operation key point to obtain a camera three-dimensional coordinate of the operation key point. According to the method, the recognition precision of the gesture key points can be improved, and then the control precision of gesture recognition interaction can be improved.

Description

Technical Field

[0001] The present application relates to the field of image acquisition technology, and in particular to a gesture key point recognition method, device, head-mounted device and computer-readable storage medium. Background Art

[0002] With the development of electronic technology and the continuous improvement of people's living standards, various smart wearable devices such as smart glasses, smart helmets and other head-mounted devices are becoming more and more popular. At present, there are many main interaction methods for smart wearable devices, such as gesture recognition interaction, eye tracking interaction, voice command interaction, interactive control device (such as ring, etc.) interaction, etc. Among them, the position recognition of gesture key points has important guiding significance for gesture recognition interaction. However, the inventor of this application found in the actual research and development process that the recognition accuracy of gesture key points in the related technology is low, which leads to low control accuracy of gesture recognition interaction. Summary of the invention

[0003] The present application provides a gesture key point recognition method, device, head-mounted device and computer-readable storage medium, which can improve the recognition accuracy of gesture key points, and further improve the control accuracy of gesture recognition interaction.

[0004] In a first aspect, the present application provides a method for identifying key points of a gesture, the method comprising:

[0005] Obtaining a first viewfinder image of the head mounted device;

[0006] Perform key point detection based on the first viewfinder image to obtain image two-dimensional coordinates of operation key points of a target gesture in the first viewfinder image and image two-dimensional coordinates of reference key points of the target gesture;

[0007] Performing projection processing based on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point to obtain second projected three-dimensional coordinates of the operation key point;

[0008] Obtaining the overall offset of the target gesture;

[0009] A conversion process is performed according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

[0010] In a second aspect, the present application provides a gesture key point recognition device, the gesture key point recognition device comprising:

[0011] A detection unit, used to obtain a first framing image of the head mounted device;

[0012] The detection unit is used to perform key point detection based on the first viewfinder image to obtain the image two-dimensional coordinates of the operation key points of the target gesture in the first viewfinder image and the image two-dimensional coordinates of the reference key points of the target gesture;

[0013] A processing unit, configured to perform projection processing based on the two-dimensional image coordinates of the operation key point and the two-dimensional image coordinates of the reference key point to obtain a second projection three-dimensional coordinate of the operation key point;

[0014] An acquisition unit, used to acquire the overall offset of the target gesture;

[0015] The conversion unit is used to perform conversion processing according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

[0016] In a third aspect, the present application also provides a head-mounted device, which includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, it executes any one of the gesture key point recognition methods provided in the present application.

[0017] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the gesture key point recognition method.

[0018] In the embodiment of the present application, on the first hand, since the image two-dimensional coordinates of the key points are difficult to accurately express the positions of the key points of the gesture in the actual three-dimensional space, by performing key point detection on the first viewfinder, the image two-dimensional coordinates of the post-operation key points and the image two-dimensional coordinates of the reference key points are obtained, and then the image two-dimensional coordinates of the operation key points and the image two-dimensional coordinates of the reference key points are used for projection processing to obtain the second projected three-dimensional coordinates of the operation key points, and the key points of the target gesture can be converted from the image two-dimensional coordinates to the three-dimensional space position, so as to restore the size of the key points of the target gesture in the camera coordinate system, thereby facilitating the acquisition of the actual space three-dimensional position of the key points and improving the recognition accuracy of the gesture key points; on the second hand, since the overall offset of the target gesture can reflect the relative deviation between the second projected three-dimensional coordinates of the key points and the camera three-dimensional coordinates of the key points, by obtaining the overall offset of the target gesture, the camera three-dimensional coordinates of the operation key points are obtained according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key points, and the camera three-dimensional coordinates of the operation key points are converted, and the camera three-dimensional coordinates of the key points can be approximated to the actual space position, so that the calculated camera three-dimensional coordinates of the operation key points are closer to the actual space position value, which can improve the recognition accuracy of the gesture key points, and thus improve the control accuracy of the gesture recognition interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a schematic structural block diagram of a head mounted device provided in an embodiment of the present application;

[0021] Figure 2 This is a flow chart of a method for identifying key points of a gesture provided by an embodiment of the present application;

[0022] Figure 3 is a schematic diagram for explaining a scene of a framing screen provided in an embodiment of the present application;

[0023] Figure 4 is a schematic diagram for explaining a scene of 21 key points of a hand provided in an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a recognition result of the image position of the key operation point in the embodiment of the present application;

[0025] Figure 6 is a scene schematic diagram of weak perspective projection coordinates of key operation points in an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of a process of adjusting and shooting a picture provided in an embodiment of the present application;

[0027] Figure 8 is a schematic diagram of a scene of the second framing screen in an embodiment of the present application;

[0028] Fig. 9 is a schematic diagram of a scene of a target framing screen in an embodiment of the present application;

[0029] Fig.10 is a schematic diagram illustrating a scenario during a translation operation in an embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the structure of an embodiment of a gesture key point recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0032] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0033] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0034] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0035] The embodiments of the present application provide a gesture key point recognition method, device, head-mounted device and computer-readable storage medium. Among them, the gesture key point recognition device can be integrated in a head-mounted device. Among them, the head-mounted device can be smart glasses, smart helmets, etc., smart glasses can be AR (augmented reality) glasses, VR (Virtual Reality) glasses, MR (Mixed Reality) glasses, XR (eXtended Reality) glasses, etc., and the smart helmet can be an AR helmet, etc. The head-mounted device is provided with a camera for capturing images, and a display module (such as lenses of smart glasses) for realizing the display of images (such as framing images, image images, etc.).

[0036] The executor of the gesture key point recognition method in the embodiment of the present application may be the gesture key point recognition device provided in the embodiment of the present application, or a head-mounted device integrating the gesture key point recognition device, wherein the gesture key point recognition device may be implemented in hardware or software.

[0037] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0038] Figure 1 It is a structural schematic block diagram of a head-mounted device provided in an embodiment of the present application.

[0039] like Figure 1 As shown, the head mounted device 100 includes a processor 101 and a memory 102, and the processor 101 and the memory 102 are connected via a bus 103, such as an I2C (Inter-integrated Circuit) bus.

[0040] Specifically, the processor 101 is used to provide computing and control capabilities to support the operation of the entire head-mounted device 100. The processor 101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0041] Specifically, the memory 102 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0042] Those skilled in the art will understand that Figure 1 The structure shown in the figure is merely a block diagram of a partial structure related to the embodiment of the present application, and does not constitute a limitation on the head-mounted device to which the embodiment of the present application is applied. The specific head-mounted device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0043] The processor 101 is used to run the computer program stored in the memory 102, and implement any one of the gesture key point recognition methods provided in the embodiments of the present application when executing the computer program. For example, the processor 101 is used to run the computer program stored in the memory 102, and can implement the following steps when executing the computer program:

[0044] Acquire a first framing picture of the head mounted device; perform key point detection based on the first framing picture to obtain the image two-dimensional coordinates of the operation key points of the target gesture in the first framing picture and the image two-dimensional coordinates of the reference key points of the target gesture; perform projection processing based on the image two-dimensional coordinates of the operation key points and the image two-dimensional coordinates of the reference key points to obtain the second projected three-dimensional coordinates of the operation key points; acquire the overall offset of the target gesture; perform conversion processing based on the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key points to obtain the camera three-dimensional coordinates of the operation key points.

[0045] It should be noted that technicians in the relevant field can clearly understand that for the convenience and brevity of description, the specific working process of the head-mounted device described above can refer to the corresponding process in the following gesture key point recognition method embodiment, and will not be repeated here.

[0046] In the following, Figure 1 The head mounted device shown in is used as an example to perform the gesture key point recognition method, and the gesture key point recognition method provided in the embodiment of the present application is described in detail. For the sake of simplicity and convenience of description, the execution subject will be omitted in the subsequent method embodiments. Figure 1 The scenario is only used to explain the gesture key point recognition method provided in the embodiment of the present application, but does not constitute a limitation on the application scenario of the gesture key point recognition method provided in the embodiment of the present application.

[0047] See also Figure 2 , Figure 2 201 to 205, wherein:

[0048] 201. Obtain a first viewing image of a head mounted device.

[0049] Among them, the head-mounted device can be a device with shooting function such as smart glasses and smart helmets. The smart glasses can be AR glasses, VR glasses, MR glasses, XR glasses, etc., and the smart helmet can be an AR helmet, etc.

[0050] Among them, the viewfinder screen is the screen when the camera of the head-mounted device is turned on but has not yet been shot.

[0051] The first framing picture is a framing picture before performing a picture adjustment operation (such as focusing, zooming, moving, etc.).

[0052] Please refer to Fig. 9 , when the head-mounted device enters the shooting mode, the field of view that the camera on the head-mounted device (such as AR glasses) can capture is the image image_ori. The head-mounted device can select the image within the field of view (i.e., on the image image_ori) and shoot to obtain the final image. Among them, the image selected within the field of view (i.e., on the image image_ori) is the viewfinder image, and the image finally captured by the head-mounted device is the same as the viewfinder image. Since the FOV (Field of View) of the camera on the head-mounted device (such as AR glasses) is much smaller than the human field of view, it cannot be guaranteed that what the human eye sees is what you want to shoot, so the camera's viewfinder image is displayed on the head-mounted device (such as AR glasses). Among them, the viewfinder image can be obtained by scaling image_ori proportionally (the specific parameters can be adjusted), and the depth value of the viewfinder image is the image depth value of the head-mounted device.

[0053] For example, Figure 3 As shown, after the user wears the head-mounted device, the user can turn on the gesture key point recognition function. When the user turns on the gesture key point recognition function, the head-mounted device enters the shooting mode, and the camera of the head-mounted device begins to capture the real scene within its field of view. The display module of the head-mounted device (such as the lenses of smart glasses) displays the image captured by the head-mounted device. At this time, the image displayed by the head-mounted device can be used as the first framing image.

[0054] 202. Perform key point detection based on the first framing picture to obtain two-dimensional image coordinates of operation key points of a target gesture in the first framing picture and two-dimensional image coordinates of reference key points of the target gesture.

[0055] Among them, reference Figure 4 , you can pre-set the gesture key points to be recognized according to the actual business scenario requirements (for example, the joint points of the five fingers such as the index finger and middle finger, and the wrist point, a total of 21 key points), collect the hand sample data of the gesture key points to train the key point detection model, and obtain the trained key point detection model. At this time, the trained key point detection model can be used to detect gesture key points (such as the thumb tip point, index finger tip point, middle finger root point, wrist point). For example, Figure 4 As shown, for the 21 key points of the hand, in this embodiment, the index finger tip point (reference Figure 4 Point 8) and the thumb tip point (reference Figure 4 Point 4) in the middle is used as the key point of operation, and the middle finger is pointed to the root point (reference Figure 4 Position point 9) and wrist point (reference Figure 4 The position point 0 in the figure is used as the reference key point, then the thumb tip point, index finger tip point, middle finger base point and wrist point can be used as gesture key points, and hand sample data of the thumb tip point, index finger tip point, middle finger base point and wrist point are collected to train the key point detection model to obtain the trained key point detection model. At this time, the trained key point detection model can be used to detect the thumb tip point, index finger tip point, middle finger base point and wrist point.

[0056] Among them, the key operation points are key points used to control screen adjustment operations (such as focus, zoom, move, etc.). The key operation points and their number can be set according to the actual business scenario requirements. For example, in some embodiments, the key operation points can be the tip of the index finger; in some embodiments, the key operation points can be the tip of the thumb; in some embodiments, the key operation points can also be the tip of the thumb and the tip of the index finger.

[0057] The reference key point is a key point used to calculate the operation depth value of the operation key point. The reference key points and their number can be set according to the actual business scenario requirements. For example, in some embodiments, the reference key points can be the wrist point and the base point of the middle finger; in some embodiments, the reference key points can be the wrist point and the base point of the index finger.

[0058] In some embodiments, in order to reduce the number of key points detected, two reference key points (i.e., the first key point and the second key point, such as the wrist point and the middle finger base point) are detected to calculate the operation depth value of the operation key point, and two operation key points (i.e., the third key point and the fourth key point, such as the thumb tip point and the index finger tip point) are detected to control the picture adjustment operation (such as focus, zoom, move, etc.). For example, 21 key points and 4 key points are respectively used for detection and comparison through the key point detection model mediapipe, as follows: the original 21-point network input size is 256*256, but when only 4 points need to be recognized, the network input size can be changed to 64*64, and the corresponding blazeblock can be reduced from the original 43 to 34 (blazeblock is the basic module of mediapipe), the amount of calculation is reduced from the original 405.51M to 21.96M, and the amount of parameters is reduced from 2M to 1M, which greatly reduces the amount of calculation. In addition, the reference Figure 3 , Figure 3 (a), (b), and (c) are the framing images of different hand gestures in the first-person perspective. In the first-person perspective, the index finger and thumb are not easily blocked compared to the middle finger, ring finger, and little finger, and the root point and wrist point of the middle finger are fixed (i.e., not as flexible as the middle finger, ring finger, and little finger). This can greatly reduce the difficulty of neural network reasoning, thereby reducing the amount of network parameters and calculation.

[0059] The image two-dimensional coordinates of the key operation points are obtained by using the key point detection model to identify the 2D coordinates of the key operation points in the first viewfinder.

[0060] The two-dimensional coordinates of the image of the reference key point are obtained by using the key point detection model to identify the 2D coordinates of the reference key point in the first viewfinder.

[0061] For example, taking "the operation key points are the thumb tip and the index finger tip, and the reference key points are the wrist point and the middle finger base point" as an example, in step 202, through the trained key point detection model, key point detection is performed based on the first framing picture, and the 2D coordinates of the thumb tip, index finger tip, middle finger base point, and wrist point in the first framing picture can be obtained (that is, the image two-dimensional coordinates of the thumb tip, the image two-dimensional coordinates of the index finger tip, the image two-dimensional coordinates of the middle finger base point, and the image two-dimensional coordinates of the wrist point), thereby obtaining the image two-dimensional coordinates of the operation key points and the image two-dimensional coordinates of the reference key points.

[0062] For example, the image 2D coordinates of the operation key points and the image 2D coordinates of the reference key points can be as follows:

[0063] The image two-dimensional coordinates of the thumb tip point (referred to as thumb tip point 2d coordinates for short): (X1, Y1).

[0064] The image two-dimensional coordinates of the index finger tip (referred to as the index finger tip 2d coordinates): (X2, Y2).

[0065] The image two-dimensional coordinates of the wrist point (referred to as wrist point 2D coordinates): (X3, Y3).

[0066] The image two-dimensional coordinates of the middle finger root point (referred to as the middle finger root point 2d coordinates): (X4, Y4).

[0067] 203. Perform projection processing based on the two-dimensional image coordinates of the operation key point and the two-dimensional image coordinates of the reference key point to obtain second projected three-dimensional coordinates of the operation key point.

[0068] The reference key points include the first key point and the second key point. There are multiple implementation methods in step 203, which include, for example, the following method 1 and method 2:

[0069] The first key point and the second key point may be set according to actual business scenario requirements. For example, the first key point may be a wrist point and the second key point may be a middle finger base point.

[0070] Method 1: Calculate the second projection 3D coordinates of the key operation points separately. This includes the following steps (1.1) to (1.3):

[0071] (1.1) Perform weak perspective projection processing on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point respectively to obtain the first projection three-dimensional coordinates of the operation key point and the first projection three-dimensional coordinates of the reference key point.

[0072] The first projection three-dimensional coordinates of the reference key point are the weak perspective projection positions of the reference key point obtained by performing a weak perspective projection to obtain the relative scale position of the reference key point, and then performing an absolute position conversion on the relative scale position of the reference key point. Similar to the operation key point, the first projection three-dimensional coordinates of the reference key point may include the first axis projection coordinate value of the reference key point, the second axis projection coordinate value of the reference key point, and the third axis projection coordinate value of the reference key point.

[0073] Exemplarily, first, based on the weak perspective projection relationship, the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point are subjected to weak perspective projection, and the 2.5D normalized result of the weak perspective projection is obtained to obtain the weak perspective projection coordinates of the operation key point (hereinafter referred to as 2.5D coordinates) and the weak perspective projection coordinates (2.5D coordinates) of the reference key point. Then, since the weak perspective projection coordinates (2.5D coordinates) reflect the relative position deviation between the two key points, that is, the weak perspective projection coordinates (2.5D coordinates) have no scale and absolute position, therefore, in this embodiment, one of the coordinate values ​​(such as Z) of the weak perspective projection coordinates (2.5D coordinates) of the operation key point is obtained. 33 ') is set to a fixed value (such as 0), so that the weak perspective projection coordinates of the operation key point (hereinafter referred to as 2.5d coordinates) and the weak perspective projection coordinates (2.5d coordinates) of the reference key point are converted into absolute positions. When one of the coordinate values ​​(such as Z 33 ') is set to a fixed value (such as 0), the first projection three-dimensional coordinates of the operation key point and the first projection three-dimensional coordinates of the reference key point can be calculated.

[0074] For ease of understanding, the example of step 202 above is continued. For example, taking "the operation key points are the thumb tip and the index finger tip, and the reference key points are the wrist point and the middle finger base point" as an example, first, the image two-dimensional coordinates (X1, Y1) of the thumb tip, the image two-dimensional coordinates (X2, Y2) of the index finger tip, the image two-dimensional coordinates (X3, Y3) of the wrist point, and the image two-dimensional coordinates (X4, Y4) of the middle finger base point are weakly perspective projected to obtain the 2.5d coordinate normalization result of the weak perspective projection, and the weak perspective projection coordinates of the thumb tip, the index finger tip, the wrist point, and the middle finger base point are obtained; thereby obtaining the weak perspective projection coordinates of the operation key points and the weak perspective projection coordinates of the reference key points. For example, the weak perspective projection coordinates of the operation key points and the weak perspective projection coordinates of the reference key points can be as follows:

[0075] The weak perspective projection coordinates of the thumb tip (referred to as the 2.5d coordinates of the thumb tip): (X 11 ',Y 11 ',Z 11 ').

[0076] The weak perspective projection coordinates of the index finger tip (referred to as the 2.5d coordinates of the index finger tip): (X 22 ',Y 22 ',Z 22 ').

[0077] The weak perspective projection coordinates of the wrist point (referred to as the wrist point 2.5d coordinates): (X 33 ',Y 33 ',Z 33 ').

[0078] The weak perspective projection coordinates of the middle finger root point (referred to as the 2.5d coordinates of the middle finger root point): (X 44 ',Y 44 ',Z 44 ').

[0079] Then, set one of the coordinate values ​​of the weak perspective projection coordinates (referred to as 2.5d coordinates in this article) of the key point to a fixed value. For example, you can set Z 33 '=0, so that the weak perspective projection coordinates of the operation key point and the weak perspective projection coordinates of the reference key point, as well as Z 33 '=0, the first projection 3D coordinates of the operation key point and the first projection 3D coordinates of the reference key point are calculated as follows:

[0080] The first projection three-dimensional coordinates of the thumb tip point: (X 11 ,Y 11 ,Z 11 ).

[0081] The first projection three-dimensional coordinates of the index finger tip: (X 22 ,Y 22 ,Z 22 ).

[0082] The first projection 3D coordinates of the wrist point: (X 33 ,Y 33 ,Z 33 ).

[0083] The first projection three-dimensional coordinates of the middle finger root point: (X 44 ,Y 44 ,Z 44 ).

[0084] (1.2) Obtain a reference distance between the first key point and the second key point.

[0085] The reference distance refers to the distance between the first key point and the second key point.

[0086] In some embodiments, the user input distance may be used as a reference distance (denoted as L) between the first key point and the second key point. In some embodiments, a preset value may be used as the reference distance L between the first key point and the second key point.

[0087] For example, taking "the first key point is the wrist point and the second key point is the base point of the middle finger" as an example, in some embodiments, the distance input by the user can be used as the distance between the wrist point and the base point of the middle finger; in some embodiments, the preset value can be used as the distance L between the wrist point and the base point of the middle finger, for example, the distance between the wrist point and the base point of the middle finger of an adult is 9.5 cm.

[0088] (1.3) Determine the second projected three-dimensional coordinates of the operation key point according to the first projected three-dimensional coordinates of the operation key point, the first projected three-dimensional coordinates of the reference key point, and the reference distance.

[0089] Since the first projected 3D coordinates of the key point mainly reflect the relative position, but this relative position may be different from the actual 3D position, it is necessary to restore the first projected 3D coordinates of the key point to the real size (for example, the detailed restoration method can refer to the description of step (1.3) or step (2.3)) to restore the key point to the real 3D size position. Among them, the second projected 3D coordinates refer to the 3D coordinates of the key point obtained by performing the real size restoration on the first projected 3D coordinates of the key point.

[0090] The second projected three-dimensional coordinates of the operation key point refer to the three-dimensional coordinates of the operation key point obtained by performing real-size restoration processing on the first projected three-dimensional coordinates of the operation key point.

[0091] Exemplarily, the first projected three-dimensional coordinates of the operation key point and the first projected three-dimensional coordinates of the first key point (such as (X 33 ,Y 33 ,Z 33 )), the first projection 3D coordinates of the second key point (such as (X 44 ,Y 44 ,Z 44 )), and the distance L between the first key point and the second key point are substituted into the preset first scale recovery calculation formula to calculate the second projection three-dimensional coordinates of the operation key point.

[0092] For example, taking "the first key point is the wrist point, the second key point is the base of the middle finger, and the operation key points include the thumb tip point and the index finger tip point" as an example, in step (1.3), the first projection 3D coordinate (X 11 ,Y 11 ,Z 11 ), the first projection three-dimensional coordinate of the index finger tip (X 22 ,Y 22 ,Z 22 ), the first projection 3D coordinate of the wrist point (X 33 ,Y 33 ,Z 33 ), the first projection three-dimensional coordinates of the middle finger root point (X 44 ,Y 44 ,Z 44 ), the distance L between the wrist point and the middle finger root point, and substitute it into the first scale recovery calculation formula shown in the following formula 1 to calculate the second projection three-dimensional coordinates of the thumb fingertip point (X R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2 ).

[0093]

[0094] In formula 1, (X 11 ,Y 11 ,Z 11 )、(X 22 ,Y 22 ,Z 22 )、(X 33 ,Y 33 ,Z 33 )、(X44 ,Y 44 ,Z 44 ) are the first projection three-dimensional coordinates of the thumb tip, index finger tip, wrist point, and middle finger root point, L is the distance between the wrist point and the middle finger root point, (X R1 ,Y R1 ,Z R1 )、(X R2 ,Y R2 ,Z R2 ) are the second projection 3D coordinates of the thumb tip and index finger tip respectively.

[0095] Method 2: Calculate the second projection 3D coordinates of the operation key point and the second projection 3D coordinates of the reference key point at the same time. It includes the following steps (2.1) to (2.3):

[0096] (2.1) Perform weak perspective projection processing on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point respectively to obtain the first projection three-dimensional coordinates of the operation key point and the first projection three-dimensional coordinates of the reference key point.

[0097] The implementation of step (2.1) is similar to that of step (1.1). For details, please refer to the relevant instructions in the previous text and will not be repeated here.

[0098] (2.2) Obtain a reference distance between the first key point and the second key point.

[0099] The implementation of step (2.2) is similar to step (1.2). For details, please refer to the relevant instructions in the previous text and will not be repeated here.

[0100] (2.3) Determine the second projected three-dimensional coordinates of the operation key point and the second projected three-dimensional coordinates of the reference key point based on the first projected three-dimensional coordinates of the operation key point, the first projected three-dimensional coordinates of the reference key point, and the reference distance.

[0101] The second projected three-dimensional coordinates of the reference key point refer to the three-dimensional coordinates of the reference key point obtained by performing true size restoration processing on the first projected three-dimensional coordinates of the reference key point.

[0102] Exemplarily, the first projected three-dimensional coordinates of the operation key point and the first projected three-dimensional coordinates of the first key point (such as (X 33 ,Y 33 ,Z 33 )), the first projection 3D coordinates of the second key point (such as (X 44 ,Y 44 ,Z 44)), and the distance L between the first key point and the second key point are substituted into the preset second scale recovery calculation formula to calculate the second projection three-dimensional coordinates of the operation key point and the second projection three-dimensional coordinates of the first key point (such as (X R3 ,Y R3 ,Z R3 )), the second projection 3D coordinates of the second key point (such as (X R4 ,Y R4 ,Z R4 )).

[0103] For example, taking "the first key point is the wrist point, the second key point is the base of the middle finger, and the operation key points include the thumb tip point and the index finger tip point" as an example, in step (1.3), the first projection 3D coordinate (X 11 ,Y 11 ,Z 11 ), the first projection three-dimensional coordinate of the index finger tip (X 22 ,Y 22 ,Z 22 ), the first projection 3D coordinate of the wrist point (X 33 ,Y 33 ,Z 33 ), the first projection three-dimensional coordinates of the middle finger root point (X 44 ,Y 44 ,Z 44 ), the distance L between the wrist point and the middle finger root point, and substitute it into the second scale recovery calculation formula shown in Formula 2 below to calculate the second projection three-dimensional coordinates of the thumb fingertip point (X R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2 ), the second projection 3D coordinates of the wrist point (X R3 ,Y R3 ,Z R3 ), the second projection three-dimensional coordinates of the middle finger root point (X R4 ,Y R4 ,Z R4 ).

[0104]

[0105] In formula 2, (X 11 ,Y 11 ,Z 11 )、(X 22 ,Y 22 ,Z 22 )、(X 33 ,Y 33 ,Z33 )、(X 44 ,Y 44 ,Z 44 ) are the first projection three-dimensional coordinates of the thumb tip, index finger tip, wrist point, and middle finger root point, L is the distance between the wrist point and the middle finger root point, (X R1 ,Y R1 ,Z R1 )、(X R2 ,Y R2 ,Z R2 )、(X R3 ,Y R3 ,Z R3 )、(X R4 ,Y R4 ,Z R4 ) are the second projection three-dimensional coordinates of the thumb tip, index finger tip, wrist point, and middle finger base point respectively.

[0106] 204. Obtain an overall offset of the target gesture.

[0107] There are many ways to implement step 204, illustratively including:

[0108] <1> Based on the key points of operation.

[0109] At this time, step 204 may specifically include: obtaining the camera intrinsic parameters of the head-mounted device; determining the overall offset of the target gesture based on the image two-dimensional coordinates of the operation key point, the second projection three-dimensional coordinates of the operation key point and the camera intrinsic parameters.

[0110] For example, taking "the key operation points include the thumb tip and the index finger tip" as an example, the image two-dimensional coordinates of the thumb tip (X1, Y1), the image two-dimensional coordinates of the index finger tip (X2, Y2), and the second projection three-dimensional coordinates of the thumb tip (X R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2 ) and the camera intrinsic parameters of the head-mounted device are substituted into the following Formula 3 and Formula 4 to solve for the overall offset (X, Y, Z) of the target gesture.

[0111] Among them, Formula 3 to Formula 6 have a total of 4 equation groups (each formula is an equation group), and the last equation of each equation group is an identity, so there are a total of 8 equations and 3 unknowns. It is an overdetermined equation group, and its least squares solution can be obtained; therefore, using two equation groups in Formulas 3 to 6, the overall offset (X, Y, Z) of the target gesture can be obtained.

[0112] <2> Based on reference key points.

[0113] At this time, step 204 may specifically include: obtaining the camera intrinsic parameters of the head-mounted device; determining the overall offset of the target gesture based on the image two-dimensional coordinates of the reference key point, the second projection three-dimensional coordinates of the reference key point and the camera intrinsic parameters.

[0114] For example, taking "the first key point is the wrist point and the second key point is the base of the middle finger" as an example, the image two-dimensional coordinates of the wrist point (X3, Y3), the image two-dimensional coordinates of the base of the middle finger (X4, Y4), and the second projection three-dimensional coordinates of the wrist point (X R3 ,Y R3 ,Z R3 ), the second projection three-dimensional coordinates of the middle finger root point (X R4 ,Y R4 ,Z R4 ) and the camera internal parameters of the head-mounted device are substituted into the following formula 5 and formula 6 to solve for the overall offset (X, Y, Z) of the target gesture.

[0115] <3> Acquired based on operation key points and reference key points.

[0116] At this time, step 204 may specifically include: obtaining the camera intrinsic parameters of the head-mounted device; determining the overall offset of the target gesture based on the image two-dimensional coordinates of the operation key point, the second projection three-dimensional coordinates of the operation key point, the image two-dimensional coordinates of the reference key point, the second projection three-dimensional coordinates of the reference key point and the camera intrinsic parameters.

[0117] For example, taking "the first key point is the wrist point, the second key point is the base point of the middle finger, and the operation key points include the thumb tip point and the index finger tip point" as an example, the image two-dimensional coordinates of the thumb tip point (X1, Y1), the image two-dimensional coordinates of the index finger tip point (X2, Y2), the image two-dimensional coordinates of the wrist point (X3, Y3), the image two-dimensional coordinates of the middle finger base point (X4, Y4), the second projection three-dimensional coordinates of the thumb tip point (X5, Y6), and the second projection three-dimensional coordinates of the thumb tip point (X7, Y8) can be obtained. R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2 ), the second projection 3D coordinates of the wrist point (X R3 ,Y R3 ,Z R3 ), the second projection three-dimensional coordinates of the middle finger root point (X R4 ,Y R4 ,Z R4) and the camera internal parameters of the head-mounted device are substituted into the following Formula 3, Formula 4, Formula 5, and Formula 6 to solve for the overall offset (X, Y, Z) of the target gesture.

[0118]

[0119] In formula 3-6, each formula is a system of equations. are the camera internal parameters of the head-mounted device, (X1, Y1)(X2, Y2)(X3, Y3)(X4, Y4) are the two-dimensional image coordinates of the thumb tip (i.e., the key point of operation), the index finger tip (i.e., the key point of operation), the wrist point (i.e., the first key point), and the middle finger root point (i.e., the second key point), respectively. R1 ,Y R1 ,Z R1 )、(X R2 ,Y R2 ,Z R2 )、(X R3 ,Y R3 ,Z R3 )、(X R4 ,Y R4 ,Z R4 ) are the second projection three-dimensional coordinates of the thumb tip (i.e., the operation key point), the index finger tip (i.e., the operation key point), the wrist point (i.e., the first key point), and the middle finger base point (i.e., the second key point), and (X, Y, Z) is the overall offset.

[0120] Since the 2.5d coordinates of the key points (i.e., the weak perspective projection coordinates of each key point, such as the weak perspective projection coordinates of the operation key point) are based on the weak perspective projection transformation, but the actual imaging of the object in the camera is based on perspective projection, there will be an error between the first projection three-dimensional coordinates obtained by the weak perspective projection processing in step (1.1) and the real perspective projection. On the one hand, the weak perspective projection coordinates (2.5d coordinates) obtained by the weak perspective projection are processed into first projection three-dimensional coordinates through step (1.1), and the second projection three-dimensional coordinates of the operation key point are determined according to the first projection three-dimensional coordinates of the operation key point, the first projection three-dimensional coordinates of the reference key point and the reference distance through steps (1.2) to (1.3), so that the "weak perspective projection coordinates of the operation key point obtained based on the weak perspective projection relationship (i.e., the 2.5d coordinates of the key point of the gesture with relatively larger error)" can be restored to the real scale, and the second projection three-dimensional coordinates of the operation key point of the real scale (i.e., the more realistic 3D coordinates of the key point of the gesture) are obtained. Secondly, step 204 first uses the image two-dimensional coordinates of the operation key point, the second projection three-dimensional coordinates of the operation key point and the camera internal parameters to determine the overall offset of the target gesture; then step 205 performs conversion based on the overall offset to obtain the camera three-dimensional coordinates of the operation key point, so that the calculated camera three-dimensional coordinates of the operation key point are closer to the actual value, thereby reducing the error between the weak perspective projection and the real perspective projection, thereby improving the calculation accuracy of the operation depth value of the operation key point.

[0121] 205 . Perform conversion processing according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

[0122] There are many ways to implement step 205, which include, for example:

[0123] <1> Determined based on the second projection 3D coordinates of the operation key points.

[0124] At this time, step 205 may specifically include: substituting the overall offset of the target gesture and the second projection three-dimensional coordinates of the operation key point into a preset first conversion formula to obtain the camera three-dimensional coordinates of the operation key point. For example, as shown in the following formula 7, taking "the operation key point includes the thumb tip point and the index finger tip point" as an example, the overall offset (X, Y, Z) of the target gesture, the second projection three-dimensional coordinates (X R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2), and substitute it into the first conversion formula shown in the following formula 7 to calculate the camera three-dimensional coordinates of the thumb fingertip point (X W1 ,Y W1 ,Z W1 ), the camera 3D coordinates of the index finger tip (X W2 ,Y W2 ,Z W2 ).

[0125]

[0126] In formula 7, (X W1 ,Y W1 ,Z W1 )、(X W2 ,Y W2 ,Z W2 ) are the camera 3D coordinates of the thumb tip and index finger tip respectively.

[0127] <2> Determined based on the second projected three-dimensional coordinates of the operation key point and the second projected three-dimensional coordinates of the reference key point.

[0128] At this time, step 205 may specifically include: substituting the overall offset of the target gesture, the second projected three-dimensional coordinates of the operation key point, the second projected three-dimensional coordinates of the first key point, and the second projected three-dimensional coordinates of the second key point into a preset second conversion formula to obtain the camera three-dimensional coordinates of the operation key point, the camera three-dimensional coordinates of the first key point, and the camera three-dimensional coordinates of the second key point. For example, as shown in the following formula 7, taking "the first key point is the wrist point, the second key point is the base point of the middle finger, and the operation key points include the thumb tip point and the index finger tip point" as an example, the overall offset (X, Y, Z) of the target gesture, the second projected three-dimensional coordinates of the thumb tip point (X R1 ,Y R1 ,Z R1 ), the second projection three-dimensional coordinates of the index finger tip (X R2 ,Y R2 ,Z R2 ), the second projection 3D coordinates of the wrist point (X R3 ,Y R3 ,Z R3 ), the second projection three-dimensional coordinates of the middle finger root point (X R4 ,Y R4 ,Z R4 ), and substitute it into the second conversion formula shown in Formula 8 below to calculate the camera three-dimensional coordinates (X W1 ,Y W1 ,Z W1 ), the camera 3D coordinates of the index finger tip (X W2 ,Y W2 ,ZW2 ), the camera 3D coordinates of the wrist point (X W3 ,Y W3 ,Z W3 ), the camera 3D coordinates of the middle finger root point (X W4 ,Y W4 ,Z W4 ).

[0129]

[0130] In formula 8, (X W1 ,Y W1 ,Z W1 )、(X W2 ,Y W2 ,Z W2 )、(X W3 ,Y W3 ,Z W3 )、(X W4 ,Y W4 ,Z W4 ) are the camera 3D coordinates of the thumb tip, index finger tip, wrist point, and middle finger base point respectively.

[0131] From the above content, it can be seen that, firstly, since the image two-dimensional coordinates of the key points are difficult to accurately express the positions of the key points of the gesture in the actual three-dimensional space, by performing key point detection on the first viewfinder, the image two-dimensional coordinates of the post-operation key points and the image two-dimensional coordinates of the reference key points are obtained, and then the image two-dimensional coordinates of the operation key points and the image two-dimensional coordinates of the reference key points are used for projection processing to obtain the second projection three-dimensional coordinates of the operation key points, and the key points of the target gesture can be converted from the image two-dimensional coordinates to the three-dimensional space position, so as to restore the size of the key points of the target gesture in the camera coordinate system, and then it is convenient to obtain the actual space three-dimensional position of the key points, and improve the recognition accuracy of the gesture key points; secondly, since the overall offset of the target gesture can reflect the relative deviation between the second projection three-dimensional coordinates of the key points and the camera three-dimensional coordinates of the key points, by obtaining the overall offset of the target gesture, the camera three-dimensional coordinates of the operation key points are obtained according to the overall offset of the target gesture and the second projection three-dimensional coordinates of the operation key points. The camera three-dimensional coordinates of the key points can be approximated to the actual space position, so that the calculated camera three-dimensional coordinates of the operation key points are closer to the actual space position value, which can improve the recognition accuracy of the gesture key points and the recognition accuracy of the gesture position change, and thus improve the control accuracy of the gesture recognition interaction. Thirdly, since the camera three-dimensional coordinates of the operation key points contain the depth information of the key points, it can include indication information of the operation depth value of the operation key points, thereby improving the recognition accuracy of the gesture key points, and further improving the control accuracy of the gesture recognition interaction.

[0132] Furthermore, in some embodiments, the image may be adjusted and then photographed using the operation depth value of the operation key point obtained based on the first viewfinder image recognition. Figure 7 At this time, the method may further include the following picture adjustment operation and shooting process of steps 701 to 704:

[0133] 701. Display the first framing image.

[0134] 702 : Perform recognition based on the first framing picture to obtain an operation depth value of an operation key point of a target gesture in the first framing picture.

[0135] There are many ways to implement step 702, illustratively including:

[0136] (1) Using the first projection three-dimensional coordinates of the operation key point obtained by the weak perspective projection relationship, the operation depth value of the operation key point is extracted. Exemplarily, at this time, step 702 may specifically include the following steps 7021A to 7023A:

[0137] 7021A. Perform key point detection based on the first framing picture to obtain image two-dimensional coordinates of operation key points of the target gesture in the first framing picture.

[0138] The implementation of step 7021A is similar to step 202, and details can be found in the following related instructions.

[0139] 7022A. Perform weak perspective projection processing on the image two-dimensional coordinates of the operation key point to obtain the first projection three-dimensional coordinates of the operation key point.

[0140] Since the weak perspective projection coordinates of the key points are the relative scale relationship between the key points, it is necessary to perform absolute position conversion processing on the weak perspective projection coordinates of the key points (for example, assigning a value of 0 to the three-dimensional coordinates of one of the key points, thereby obtaining the first projection three-dimensional coordinates of one of the key points, and calculating the first projection three-dimensional coordinates of other key points based on their own weak perspective projection coordinates and the first projection three-dimensional coordinates of the assigned key points) so that the key points have absolute positions. Among them, the first projection three-dimensional coordinates refer to the three-dimensional coordinates of the key points obtained by performing absolute position conversion processing on the weak perspective projection coordinates of the key points.

[0141] The first projection three-dimensional coordinates of the key operation point are the weak perspective projection positions of the key operation points obtained by performing a weak perspective projection to obtain the relative scale positions of the key operation points, and then performing an absolute position conversion on the relative scale positions of the key operation points. The first projection three-dimensional coordinates of the key operation points include the first axis projection coordinate value of the key operation points, the second axis projection coordinate value of the key operation points, and the third axis projection coordinate value of the key operation points.

[0142] The implementation of step 7022A is similar to step (1.1), and details can be found in the following related instructions.

[0143] 7023A. Use the third-axis projection coordinate value of the operation key point as the operation depth value of the operation key point.

[0144] (2)Reference Figure 6 , Figure 6 It is a scene diagram of the weak perspective projection coordinates of the operation key points. Since the weak perspective projection coordinates (2.5d coordinates) of the operation key points are obtained by weak perspective projection, there will be errors with the real perspective projection, that is, the weak perspective projection coordinates (2.5d coordinates) are 3D coordinates with low scale accuracy. If the weak perspective projection coordinates (2.5d coordinates) of the operation key points are directly used to extract the operation depth value of the operation key points, the result will be inaccurate. In order to improve the accuracy of the operation depth value of the operation key points, the weak perspective projection coordinates (2.5d coordinates) of the operation key points obtained based on the weak perspective projection relationship are further scaled to obtain scale-restored 3D coordinates; the scale-restored 3D coordinates (that is, the three-dimensional coordinates of the camera obtained in step 205) are used to extract the operation depth value of the operation key points. Exemplarily, at this time, step 702 can specifically include the following step 7021B:

[0145] 7021B. Based on the camera three-dimensional coordinates of the operation key point, obtain the operation depth value of the operation key point.

[0146] The camera three-dimensional coordinates of the key operation point include the first axis camera coordinate value of the key operation point, the second axis camera coordinate value of the key operation point, and the third axis camera coordinate value of the key operation point. For example, the third axis camera coordinate value of the key operation point can be used as the operation depth value of the key operation point.

[0147] Therefore, through the above steps 202 to 205 and step 7021B, the 3D coordinates of the key points of the target gesture can be restored to the same size as the actual gesture (for example, based on the reference distance L from the wrist point to the base of the middle finger, the 2.5D coordinates of the key points of the gesture are restored to the same size as the actual gesture), thereby improving the fit between the 3D coordinates of the key points and the actual spatial position, thereby improving the accuracy of gesture recognition and the accuracy of gesture position changes, and thereby improving the control accuracy of screen adjustment operations (such as focus, zoom, move, etc.).

[0148] 703. Update the first framing picture according to the operation depth value of the operation key point and the picture depth value of the head mounted device to obtain an updated target framing picture.

[0149] The target framing picture is the framing picture after a picture adjustment operation (such as focusing, zooming, moving, etc.) is performed.

[0150] The image depth value (denoted as H) is the depth value of the viewfinder image of the head mounted device. The image depth value is a preset value, and the specific value of the image depth value can be set according to actual business needs, for example, within a range that can be easily touched by a human hand, such as about 40 cm.

[0151] There are many ways to update the viewfinder image in step 703, illustratively including:

[0152] (1) Focusing the viewfinder image. At this time, step 703 may specifically include the following steps 7031A to 7033A:

[0153] 7031A. When the operation depth value of the operation key point increases from small to large until it is equal to the picture depth value, the image position of the operation key point is obtained.

[0154] The image position of the key operation point is the 2D coordinate of the key operation point (such as the tip of the index finger, the tip of the thumb, etc.) in the image. Figure 5 , Figure 5 This is a schematic diagram of the recognition result of the image position of the key operation point. Figure 5 Each black dot in the diagram represents an operation key.

[0155] Exemplarily, step 7031A “obtaining the image position of the key operation point” may specifically include the following steps A1 to A3:

[0156] A1. Obtain the camera three-dimensional coordinates of the operation key point, wherein the camera three-dimensional coordinates include the first-axis camera coordinate value of the operation key point, the second-axis camera coordinate value of the operation key point, and the third-axis camera coordinate value of the operation key point.

[0157] The method for obtaining the camera 3D coordinates of the key operation points in step A1 is similar to the method for obtaining the camera 3D coordinates of the key operation points in steps 202 to 205. For details, please refer to the relevant description above and will not be repeated here.

[0158] A2. Obtain camera coordinates of a first corner point of the first viewfinder image and camera coordinates of a second corner point of the first viewfinder image.

[0159] Among them, the first corner point and the second corner point are two corner points of the first framing picture. For example, the first corner point and the second corner point can be the upper left corner point and the lower right corner point of the first framing picture respectively; for another example, the first corner point and the second corner point can be the upper right corner point and the lower left corner point of the first framing picture respectively; for another example, the first corner point and the second corner point can be the upper right corner point and the lower right corner point of the first framing picture respectively; for another example, the first corner point and the second corner point can be the upper left corner point and the lower left corner point of the first framing picture respectively; for another example, the first corner point and the second corner point can be the upper left corner point and the upper right corner point of the first framing picture respectively; for another example, the first corner point and the second corner point can be the lower left corner point and the lower right corner point of the first framing picture respectively.

[0160] The first corner point camera coordinates are the coordinates of the first corner point in the first viewfinder image.

[0161] The second corner point camera coordinates are the coordinates of the second corner point in the first viewfinder image.

[0162] For better understanding, some coordinates involved in this embodiment and their corresponding coordinate system conversion relationships are introduced below:

[0163] 1. Pixel coordinate system: a two-dimensional coordinate system. Both the pixel coordinate system and the image coordinate system are on the imaging plane. The difference is that the pixel coordinate system uses row and column values ​​to represent the coordinate values ​​of the pixel points. Among them, the pixel coordinate refers to the coordinate in the pixel coordinate system.

[0164] 2. Image coordinate system: It is a two-dimensional coordinate system. The origin is the corner point of the camera optical axis and the imaging plane. The image coordinate system and the camera coordinate system are in a perspective projection relationship. The image coordinate system is a two-dimensional coordinate system obtained by projecting the three-dimensional camera coordinate system. Among them, the image coordinates refer to the coordinates in the image coordinate system.

[0165] 3. Camera coordinate system: It is a three-dimensional coordinate system, and the origin is the light point of the camera. The camera coordinates refer to the coordinates in the camera coordinate system.

[0166] Among them, the first axis camera coordinate value, the second axis camera coordinate value, and the third axis camera coordinate value are respectively the X c Axis coordinate value, Y c Axis coordinate value, Z c Axis coordinate values.

[0167] Therefore, since the pixel coordinates of the first corner point and the second corner point in the first framing picture are known values, and the depth value of the first framing picture is equal to the picture depth value H (that is, the third-axis camera coordinate value of the first corner point = the picture depth value H, and the third-axis camera coordinate value of the second corner point = the picture depth value H), combined with the above conversion relationship between the pixel coordinate system and the image coordinate system, and the conversion relationship between the image coordinate system and the camera coordinate system, it can be obtained that the coordinates of the first corner point and the second corner point in the camera coordinate system of the head-mounted device are: (x1_image, y1_image, H) and (x2_image, y2_image, H), respectively.

[0168] Among them, the first corner point camera coordinates include the first axis camera coordinate value of the first corner point (such as x1_image here), the second axis camera coordinate value of the first corner point (such as y1_image here), and the third axis camera coordinate value of the first corner point (such as H here).

[0169] The second corner point camera coordinates include the first-axis camera coordinate value of the second corner point (such as x2_image here), the second-axis camera coordinate value of the second corner point (such as y2_image here), and the third-axis camera coordinate value of the second corner point (such as H here).

[0170] A3. Perform normalization processing based on the first corner point camera coordinates, the second corner point camera coordinates, the first axis camera coordinate value of the operation key point, and the second axis camera coordinate value of the operation key point to obtain the image position of the operation key point.

[0171] Among them, taking the operation key point as the index finger tip as an example, assuming that the camera three-dimensional coordinates of the index finger tip point obtained in step A1 are (index_x, index_y, H), then the first-axis camera coordinate value of the index finger tip point is index_x, the second-axis camera coordinate value of the index finger tip point is index_y, and the third-axis camera coordinate value of the index finger tip point is H.

[0172] Exemplarily, first, the camera coordinates of the first corner point, the camera coordinates of the second corner point, the first-axis camera coordinate value of the operation key point and the second-axis camera coordinate value of the operation key point are normalized to obtain the normalized position of the operation key point; the first-axis camera coordinate value of the first corner point (such as x1_image), the first-axis camera coordinate value of the second corner point (such as x2_image) and the first-axis camera coordinate value of the operation key point (such as index_x) are normalized to obtain the first-axis normalized value of the operation key point (such as index_x'); the second-axis camera coordinate value of the first corner point (such as y1_image), the second-axis camera coordinate value of the second corner point (such as y2_image) and the second-axis camera coordinate value of the operation key point (such as index_y) are normalized to obtain the second-axis normalized value of the operation key point (such as index_y'), and the normalized position of the operation key point can be (index_x', index_y'). Then, the normalized position (index_x′, index_y′) of the operation key point is projected onto the first viewfinder image to obtain the image position (denoted as P) of the operation key point.

[0173] 7032A. Obtain a target object distance of the first framing image according to the image position of the operation key point.

[0174] For example, taking the index finger tip as the key operation point, the image position of the index finger tip at this time can be calculated (that is, the index finger tip is located at the 2d point position in the first framing image), and with the image position of the index finger tip as the center, image image1 with a preset range of M is intercepted, and the optimal object distance is calculated with image1 as the target, and then the entire first framing image is automatically focused.

[0175] 7033A. Focus the first framing image according to the target object distance to obtain an updated target framing image.

[0176] (2) Zooming the viewfinder image. At this time, step 703 may specifically include the following steps 7031B to 7034B:

[0177] 7031B. Detect the pinch state of the left hand gesture based on the operation depth value of the left hand key point and the screen depth value.

[0178] Exemplarily, the left hand key point includes a first preset point of the left hand (such as the fingertip point of the left thumb) and a second preset point of the left hand (such as the fingertip point of the left index finger). First, it is detected whether the operation depth value of the first preset point of the left hand (such as the fingertip point of the left thumb) is less than the screen depth value H, and the operation depth value of the second preset point of the left hand (such as the fingertip point of the left index finger) is greater than the screen depth value H; then, when the operation depth value of the first preset point of the left hand (such as the fingertip point of the left thumb) is less than the screen depth value H, and the operation depth value of the second preset point of the left hand (such as the fingertip point of the left index finger) is greater than the screen depth value H, when the first preset point and the second preset point start to move, it is detected whether the distance between the first preset point of the left hand and the second preset point of the left hand (denoted as M) is satisfied. L ) is less than the preset distance threshold (denoted as M1); if the distance M between the first preset point on the left hand and the second preset point on the left hand L If the operation depth value of the first preset point of the left hand (such as the tip of the left thumb) is greater than the screen depth value H, the operation depth value of the second preset point of the left hand (such as the tip of the left index finger) is less than the screen depth value H, or the distance M between the first preset point of the left hand and the second preset point of the left hand is less than the preset distance threshold M1, then the left hand gesture is determined to be pinched. L If the distance is greater than or equal to the preset distance threshold M1, it is determined that the left hand gesture is not pinched.

[0179] 7032B. Detect the pinch state of the right hand gesture based on the operation depth value of the right hand key point and the screen depth value.

[0180] Exemplarily, the right-hand key point includes a first preset point of the right hand (such as the fingertip point of the right thumb) and a second preset point of the right hand (such as the fingertip point of the right index finger). First, it is detected whether the operation depth value of the first preset point of the right hand (such as the fingertip point of the right thumb) is less than the screen depth value H, and the operation depth value of the second preset point of the right hand (such as the fingertip point of the right index finger) is greater than the screen depth value H; then, when the operation depth value of the first preset point of the right hand (such as the fingertip point of the right thumb) is less than the screen depth value H, and the operation depth value of the second preset point of the right hand (such as the fingertip point of the right index finger) is greater than the screen depth value H, when the first preset point and the second preset point start to move, it is detected whether the distance between the first preset point of the right hand and the second preset point (denoted as M) is satisfied. R ) is less than the preset distance threshold M1; if the distance M between the first preset point on the right hand and the second preset point on the right hand R If the operation depth value of the first preset point of the right hand (such as the tip of the thumb of the right hand) is greater than the screen depth value H, the operation depth value of the second preset point of the right hand (such as the tip of the index finger of the right hand) is less than the screen depth value H, or the distance M between the first preset point of the right hand and the second preset point of the right hand is less than the preset distance threshold M1, then the right hand gesture is determined to be pinched. R If the distance is greater than or equal to the preset distance threshold M1, it is determined that the right hand gesture is not pinched.

[0181] 7033B. When the left hand gesture is pinched in and the right hand gesture is pinched in, a framing interest frame is superimposed on the first framing screen according to the current image position of the left hand gesture and the current image position of the right hand gesture to obtain a second framing screen.

[0182] Among them, the image position of the left hand gesture (denoted as L X ,L Y ) is the image position of the pinch point of the left hand gesture. For example, the pinch point of the left hand gesture may be the center point between the first preset point of the left hand (such as the fingertip of the left thumb) and the second preset point of the left hand (such as the fingertip of the left index finger). Exemplarily, the process of determining the image position of the left hand gesture may be as follows: steps a1 to a3:

[0183] a1. Obtain the image position of the first preset point of the left hand (such as the tip of the left thumb).

[0184] The method of obtaining the "image position of the first preset point on the left hand" is similar to the method of "obtaining the image position of the key operation point" in step 7031A. For details, please refer to the relevant description above and will not be repeated here. Specifically, the "key operation point" in step 7031A can be replaced with the "first preset point on the left hand" for calculation to obtain the image position of the first preset point on the left hand.

[0185] a2. Obtain the image position of the second preset point of the left hand (such as the tip of the left index finger).

[0186] The method of obtaining the "image position of the second preset point of the left hand" is similar to the method of "obtaining the image position of the key operation point" in step 7031A. For details, please refer to the relevant description above and will not be repeated here. Specifically, the "key operation point" in step 7031A can be replaced with the "second preset point of the left hand" for calculation to obtain the image position of the second preset point of the left hand.

[0187] a3. Based on the image position of the first preset point of the left hand (such as the tip of the left thumb) and the image position of the second preset point of the left hand (such as the tip of the left index finger), calculate the image position of the center point between the first preset point of the left hand and the second preset point of the left hand as the image position of the left hand gesture.

[0188] Among them, the current image position of the left hand gesture (denoted as L X1 ,L Y1 ) refers to the image position of the left hand gesture when the left hand gesture is pinched.

[0189] Among them, the image position of the right hand gesture (denoted as R X ,R Y) is the image position of the pinch point of the right hand gesture. For example, the pinch point of the right hand gesture may be the center point between the first preset point of the right hand (such as the fingertip of the right thumb) and the second preset point of the right hand (such as the fingertip of the right index finger). Exemplarily, the process of determining the image position of the right hand gesture may be as follows: steps b1 to b3:

[0190] b1. Obtain the image position of the first preset point of the right hand (such as the tip of the right thumb).

[0191] The method of obtaining the "image position of the first preset point of the right hand" is similar to the method of "obtaining the image position of the key operation point" in step 7031A. For details, please refer to the relevant description above and will not be repeated here. Specifically, the "key operation point" in step 7031A can be replaced with the "first preset point of the right hand" for calculation to obtain the image position of the first preset point of the right hand.

[0192] b2. Obtain the image position of the second preset point of the right hand (such as the tip of the right index finger).

[0193] The method of obtaining the "image position of the second preset point of the right hand" is similar to the method of "obtaining the image position of the key operation point" in step 7031A. For details, please refer to the relevant description above and will not be repeated here. Specifically, the "key operation point" in step 7031A can be replaced with the "second preset point of the right hand" for calculation to obtain the image position of the second preset point of the right hand.

[0194] b3. Based on the image position of the first preset point of the right hand (such as the tip of the right thumb) and the image position of the second preset point of the right hand (such as the tip of the right index finger), calculate the image position of the center point between the first preset point of the right hand and the second preset point of the right hand as the image position of the right hand gesture.

[0195] Among them, the current image position of the right hand gesture (denoted as R X1 ,R Y1 ) refers to the image position of the right hand gesture when the right hand gesture is pinched.

[0196] For example, Figure 8 As shown, in step 7033B, the current image position (L X1 ,L Y1 ) as the upper left corner of the framing interest box and the current image position of the right hand gesture (R X1 ,R Y1 ) is used as the lower right corner point of the framing interest box, and the framing interest box is superimposed on the first framing picture to obtain the second framing picture.

[0197] 7034B. Scaling the second framing image according to the image position of the left-hand gesture after movement and / or the image position of the right-hand gesture after movement to obtain an updated target framing image.

[0198] Among them, the current image position of the left hand gesture (denoted as L X2 ,L Y2 ) refers to the image position of the left hand gesture after the left hand gesture moves when the left hand gesture is pinched and moved. The method of obtaining the “image position of the left hand gesture after movement” is similar to the step a1 to a3 “process of determining the image position of the left hand gesture”. For details, please refer to the relevant description in the previous text, which will not be repeated here.

[0199] Among them, the current image position of the right hand gesture (denoted as R X2 ,R Y2 ) refers to the image position of the right hand gesture after the right hand gesture moves when the right hand gesture is pinched and moved. The method of obtaining the “image position of the right hand gesture after movement” is similar to the step b1 to b3 “the process of determining the image position of the right hand gesture”. For details, please refer to the relevant description in the previous text, which will not be repeated here.

[0200] Exemplarily, step 7034B may specifically include the following steps B1 to B2:

[0201] B1. Determine the size change of the framing interest frame and the scaling ratio of the second framing screen according to the image position after the left hand gesture moves and / or the image position after the right hand gesture moves.

[0202] The image position after the left hand gesture is the image position of the left hand gesture after the left hand gesture is pinched and moved.

[0203] The image position after the movement of the right-hand gesture is the image position of the right-hand gesture after the right-hand gesture is pinched and moved.

[0204] For example, Figure 8 and Fig. 9 As shown, when the left hand gesture is pinched and moved, and the right hand gesture is pinched and moved, the size change of the framing interest box and the zoom ratio of the second framing screen are calculated based on the change between the image position after the movement of the left hand gesture and the current image position of the left hand gesture, and the change between the image position after the movement of the right hand gesture and the current image position of the right hand gesture.

[0205] For example, when the left hand gesture is pinched and moved and the right hand gesture remains unchanged, the size change of the framing interest frame and the zoom ratio of the second framing screen are calculated based on the change between the image position after the left hand gesture moves and the current image position of the left hand gesture.

[0206] For example, when the right-hand gesture is pinched and moved and the left-hand gesture remains unchanged, the size change of the framing interest frame and the zoom ratio of the second framing screen are calculated based on the change between the image position after the right-hand gesture moves and the current image position of the right-hand gesture.

[0207] B2. Scaling the second framing picture according to the size change amount and the scaling ratio to obtain an updated target framing picture.

[0208] (3) Shift the viewfinder image.

[0209] (3.1) Two pinch gestures appear on the viewfinder screen.

[0210] At this time, step 703 may specifically include the following steps 7031C to 7034C:

[0211] 7031C. Detect the pinch state of the left hand gesture based on the operation depth value of the left hand key point and the screen depth value.

[0212] 7032C. Detect the pinch state of the right hand gesture based on the operation depth value of the right hand key point and the screen depth value.

[0213] 7033C. When the left hand gesture is pinched in and the right hand gesture is pinched in, a framing interest frame is superimposed on the first framing screen according to the current image position of the left hand gesture and the current image position of the right hand gesture to obtain a second framing screen.

[0214] The implementation of steps 7031C to 7033C is similar to that of steps 7031B to 7033B. For details, please refer to the relevant descriptions in the previous text, which will not be repeated here.

[0215] 7034C. According to the image position of the left-hand gesture after movement or the image position of the right-hand gesture after movement, the second framing image is translated to obtain an updated target framing image.

[0216] There are many ways to implement step 7034C, illustratively, including the following methods ① and ②

[0217] Method ①: Right hand gesture to move horizontally

[0218] At this time, step 7034C may specifically include the following steps C1 to C3:

[0219] C1. After the left hand gesture is released from pinching, when the right hand gesture is pinched and moved, obtaining the image position of the right hand gesture after moving.

[0220] C2. Determine a picture translation direction and a picture translation amount of the second framing picture according to the image position of the right-hand gesture after movement.

[0221] C3. According to the picture translation direction and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

[0222] For example, refer to Figure 8 and Fig.10 First, when the left hand gesture pinch and the right hand gesture pinch are detected in step 7033C, according to the current image position of the left hand gesture (such as L X1 ,L Y1 ) and the current image position of the right hand gesture (such as (R X1 ,R Y1 )), superimpose the framing interest frame on the first framing picture to obtain a second framing picture, such as Figure 8 Then, after the left hand gesture releases the pinch, when the right hand gesture moves, as shown Fig.10 As shown, assuming that the right-hand gesture is represented by the current image position of the right-hand gesture (such as (R X1 ,R Y1 ),refer to Fig.10 (a)) pinch and move left to the image position after the right hand gesture (as shown in (R X2 ,R Y2 ),refer to Fig.10 (b)); then according to the current image position of the right hand gesture (such as (R X1 ,R Y1 )) and the image position after the right hand gesture is moved (such as (R X2 ,R Y2 )), calculate the image translation direction and the image translation amount of the second viewfinder image. Finally, according to the image translation direction and the image translation amount of the second viewfinder image, translate the second viewfinder image to obtain an updated target scene image.

[0223] Method ②: Left hand gesture translation

[0224] At this time, step 7034C may specifically include the following steps D1 to D3:

[0225] D1. After the right-hand gesture is released from pinching, when the left-hand gesture is pinched and moved, the image position of the left-hand gesture after movement is acquired.

[0226] D2. Determine a picture translation direction and a picture translation amount of the second framing picture according to the image position of the left-hand gesture after movement.

[0227] D3. According to the picture translation direction and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

[0228] The implementation of steps D1 to D3 is similar to that of steps C1 to C3. For details, please refer to the relevant descriptions above and will not be repeated here.

[0229] (3.2) A pinch gesture appears on the viewfinder screen.

[0230] At this time, step 703 may specifically include the following steps 7031D to 7034D:

[0231] 7031D. Detect the pinch state of the target gesture based on the operation depth value of the operation key point and the screen depth value.

[0232] The implementation of step 7031D is similar to step 7031B and step 7032B. For details, please refer to the relevant instructions in the previous text and will not be repeated here.

[0233] 7032D. When the target gesture is pinched, obtain the current image position of the target gesture.

[0234] The image position of the target gesture is the image position of the pinch point of the target gesture. For example, the pinch point of the target gesture can be the center point between the first preset point of the target gesture (such as the fingertip of the left thumb) and the second preset point of the target gesture (such as the fingertip of the left index finger). The determination method of the "image position of the target gesture" is similar to steps a1 to a3 and steps b1 to b3. For details, please refer to the relevant instructions in the previous text, which will not be repeated here.

[0235] The current image position of the target gesture refers to the image position of the target gesture when the target gesture is pinched.

[0236] 7033D. When the target gesture is pinched and moved, obtain the image position of the target gesture after movement.

[0237] The image position after the target gesture is moved is the image position of the target gesture after the target gesture is pinched and moved.

[0238] The method for determining the “position of the image after the target gesture is moved” is similar to steps a1 to a3 and steps b1 to b3. For details, please refer to the relevant descriptions in the previous text, which will not be repeated here.

[0239] 7034D. Determine a picture translation direction and a picture translation amount of the first framing picture according to a current image position of the target gesture and a moved image position of the target gesture.

[0240] 7035D: Perform translation processing on the first framing picture according to the picture translation direction and the picture translation amount of the first framing picture to obtain an updated target scene picture.

[0241] The implementation of steps 7034D to 7035D is similar to that of steps C2 to C3. For details, please refer to the relevant descriptions in the previous text and will not be repeated here.

[0242] 704. Perform shooting processing based on the target framing picture.

[0243] After the screen adjustment operations (such as focus, zoom, move, etc.) in step 703 above are processed, if the target framing screen seen by the user is the screen that the user wants to shoot, the shooting can be performed directly, for example, using voice (such as saying "shoot"), touching gesture key point recognition controls (such as AR glasses can touch the temples to shoot), touching the ring if there is one, etc., and the specific method is not limited.

[0244] In order to better understand the embodiments of the present application, the following example of "the head-mounted device is an AR glasses, the index finger is used to focus, the left and right hands are pinched and moved simultaneously to zoom, and one hand is pinched to translate" is used to illustrate the gesture key point recognition process in this embodiment; wherein, the left hand pinch is when the distance between the left hand thumb tip and the left hand index finger tip is less than the preset distance M1, and the right hand pinch is when the distance between the right hand thumb tip and the right hand index finger tip is less than the preset distance M1. Please refer to Figures 8 to 10 ,The gesture key point recognition process is as follows:

[0245] (I) Enter the shooting mode. The picture taken by the camera is image_ori. Since the FOV of the camera on the AR glasses is much smaller than the human field of view, it cannot be guaranteed that what the human eye sees is what it wants to shoot. Therefore, the picture image taken by the camera is displayed on the AR glasses. Image is obtained by scaling image_ori. The specific parameters can be adjusted. The picture depth value is H (H is the preset value, which is set within the range that can be easily touched by human hands, such as about 40cm).

[0246] (ii) If the image the user sees is the image they want to shoot, they can shoot directly, for example, by using voice (such as saying "shoot"), touching the temples, or touching the rings if there are any, in any way.

[0247] (3) If you are not satisfied with the presented image, you can perform operations such as auto focus, zoom, pan, etc.;

[0248] <1> , Autofocus:

[0249] When the area of ​​interest is unclear, the user can use fingertip focus as follows:

[0250] <1.1> Extend the index finger of your left or right hand to perform hand detection and 3D fingertip point recognition. At this time, the depth of the initial fingertip point must be less than H. If it is greater than H, you need to reduce the depth of the hand to make it less than H.

[0251] Among them, 3D fingertip point recognition, that is, obtaining the camera three-dimensional coordinates of the fingertip point, at this time, the obtained camera three-dimensional coordinates of the fingertip point include the operation depth value H of the fingertip point. "3D fingertip point recognition" is similar to the above steps 202 to 205, and the details can be referred to the above related descriptions, which will not be repeated here.

[0252] <1.2>, select the part you want to focus on and click it. When the depth of the index finger tip changes from small to large, H, calculate the normalized 2D point position P of the index finger tip in the image based on the camera 3D coordinates of the finger tip:

[0253] It is known that in the camera coordinate system, the coordinates of the upper left and lower right points of the image are (x1_image, y1_image, H) and (x2_image, y2_image, H), and the coordinates of the index fingertip are (index_x, index_y, H). Since the depth values ​​of the image and the fingertip are the same at this time, the normalized P = ((index_x-x1_image) / (x2_image-x1_image), (index_y-y1_image) / (y2_image-y1_image)).

[0254] <1.3>, project the normalized position P onto image_camera, and capture the image image1 with a preset range of M with the projected position as the center. Calculate the optimal object distance with image1 as the target, and then perform autofocus on the entire image. Common methods include contrast detection autofocus, etc.;

[0255] Among them, in <1.2> to <1.3>, the process of "obtaining the projected position (i.e., the camera three-dimensional coordinates of the operation key point) according to the camera three-dimensional coordinates of the fingertip point (i.e., the camera three-dimensional coordinates of the operation key point)" is the process of calculating the image position of the operation key point. For details, please refer to the relevant instructions of steps A1 to A3 above.

[0256] <2> , Zoom:

[0257] The user uses both hands to zoom in and out on a region of interest (ROI):

[0258] <2.1>, when two hands appear, identify the 3D points of the thumb and index finger of both hands;

[0259] <2.2>, taking the left hand as an example, Figure 8 As shown, initially, the thumb tip and the index finger tip are not in a pinching state. At this time, the thumb tip is moved to a distance less than H, and the index finger tip is greater than H. At this time, pinching begins. When the distance between the two points is less than the preset value M1, it is considered that the pinching is completed. At this time, the center point of the two points is taken as the point at the upper left corner of the ROI, assuming (L X1 ,L Y1 ), similarly, the projection of the right hand on the image when pinching is taken as the point at the lower right corner of the ROI, assuming it is (R X1 ,R Y1 );

[0260] like Figures 8 to 10 As shown, the image_ori frame is the field of view, the image frame is the range of the framing screen (that is, the range of the photograph taken, fixed), and the ROI frame is the framing interest frame.

[0261] <2.3>, keep your hands together and use them to operate the ROI. Suppose at a certain moment (such as Fig. 9 ), the left hand pinch point position becomes (L X2 , L Y2 ), the left hand pinch point position becomes (R X2 , R Y2 ), making the ROI box larger. Similarly, the objects within the field of view of image_ori will also become larger accordingly, but the image box is fixed, so the captured image is still within the image box.

[0262] <2.4> After zooming is completed, when both hands are released, no further operations can be performed on the image. When one hand is released, the following translation operations can be performed.

[0263] <3> Pan:

[0264] The user pans the image using one hand:

[0265] <3.1>, the implementation of one-handed pinching is the same as zooming, and can be used with either hand;

[0266] <3.2>, such as Fig.10 As shown, you can pinch the image_ori frame to translate. The image frame is the range of the photo taken, which is still fixed.

[0267] <3.3>, such as Fig.10 As shown, move the image to where you want it, then release your finger (as shown in Fig.10 At this point, the translation is complete and no further operations are required.

[0268] After a series of operations such as focusing, zooming, and panning, the user can start shooting when the picture he sees is the one he wants to shoot.

[0269] Therefore, on the first aspect, by identifying the first framing screen based on the head-mounted device, the operation depth value of the operation key point of the target gesture in the first framing screen is obtained, and then the first framing screen is updated using the operation depth value of the operation key point and the screen depth value of the head-mounted device, so that during the shooting process, the framing screen can be focused, zoomed, and other image adjustment operations using gestures, thereby improving the image shooting quality; on the second aspect, since the first framing screen is updated using the operation depth value of the operation key point and the screen depth value of the head-mounted device, the depth information of the operation key point can be mined to perform image adjustment operations on the framing screen, thereby improving the control accuracy of the image adjustment operation, and thereby improving the image shooting quality to a certain extent.

[0270] In addition, in order to better implement the gesture key point recognition method in the embodiment of the present application, based on the gesture key point recognition method, the embodiment of the present application also provides a gesture key point recognition device, such as Fig.11 FIG. 1 is a schematic diagram of a structure of a gesture key point recognition device according to an embodiment of the present application. The gesture key point recognition device 1100 includes:

[0271] The detection unit 1101 is used to obtain a first framing picture of the head mounted device;

[0272] The detection unit 1101 is used to perform key point detection based on the first viewfinder image to obtain the image two-dimensional coordinates of the operation key points of the target gesture in the first viewfinder image and the image two-dimensional coordinates of the reference key points of the target gesture;

[0273] A processing unit 1102 is used to perform projection processing based on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point to obtain a second projection three-dimensional coordinate of the operation key point;

[0274] An acquisition unit 1103 is used to acquire the overall offset of the target gesture;

[0275] The conversion unit 1104 is used to perform conversion processing according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

[0276] In some embodiments, the acquiring unit 1103 is used to:

[0277] Obtaining camera intrinsic parameters of the head mounted device;

[0278] Based on the image two-dimensional coordinates of the operation key point, the second projection three-dimensional coordinates of the operation key point and the camera intrinsic parameter, the overall offset of the target gesture is determined.

[0279] In some embodiments, the reference key point includes a first key point and a second key point, and the processing unit 1102 is used to:

[0280] Performing weak perspective projection processing on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point respectively to obtain the first projection three-dimensional coordinates of the operation key point and the first projection three-dimensional coordinates of the reference key point;

[0281] Acquire a reference distance between the first key point and the second key point;

[0282] The second projected three-dimensional coordinates of the operation key point are determined according to the first projected three-dimensional coordinates of the operation key point, the first projected three-dimensional coordinates of the reference key point, and the reference distance.

[0283] In some embodiments, the gesture key point recognition device further includes a shooting unit (not shown in the figure), which is used to:

[0284] Displaying the first framing picture;

[0285] Based on the camera three-dimensional coordinates of the key operation point, obtaining the operation depth value of the key operation point;

[0286] According to the operation depth value of the operation key point and the screen depth value of the head mounted device, the first framing screen is updated to obtain an updated target framing screen;

[0287] A shooting process is performed based on the target framing picture.

[0288] In some embodiments, the camera three-dimensional coordinates of the key operation point include the first-axis camera coordinate value of the key operation point, the second-axis camera coordinate value of the key operation point, and the third-axis camera coordinate value of the key operation point; the shooting unit is used to:

[0289] The third-axis camera coordinate value of the operation key point is used as the operation depth value of the operation key point.

[0290] In some embodiments, the photographing unit is used to:

[0291] When the operation depth value of the operation key point increases from small to large until it is equal to the picture depth value, acquiring the image position of the operation key point;

[0292] According to the image position of the key operation point, obtaining the target object distance of the first framing picture;

[0293] Focus processing is performed on the first framing picture according to the target object distance to obtain an updated target framing picture.

[0294] In some embodiments, the photographing unit is used to:

[0295] Acquire the camera three-dimensional coordinates of the key operation point, wherein the camera three-dimensional coordinates include the first-axis camera coordinate value of the key operation point, the second-axis camera coordinate value of the key operation point, and the third-axis camera coordinate value of the key operation point;

[0296] Obtaining camera coordinates of a first corner point of the first framing picture and camera coordinates of a second corner point of the first framing picture;

[0297] Normalization processing is performed based on the first corner point camera coordinates, the second corner point camera coordinates, the first axis camera coordinate value of the operation key point, and the second axis camera coordinate value of the operation key point to obtain the image position of the operation key point.

[0298] In some embodiments, the target gesture includes a left-hand gesture and a right-hand gesture, and the operation key points include a left-hand key point of the left-hand gesture and a right-hand key point of the right-hand gesture; the shooting unit is used to:

[0299] Detecting a pinch state of the left-hand gesture based on the operation depth value of the left-hand key point and the screen depth value;

[0300] Detecting a pinch state of the right hand gesture based on the operation depth value of the right hand key point and the screen depth value;

[0301] When the left hand gesture is pinched and the right hand gesture is pinched, a framing interest frame is superimposed on the first framing picture according to the current image position of the left hand gesture and the current image position of the right hand gesture to obtain a second framing picture;

[0302] The second framing picture is updated according to the image position of the left-hand gesture after movement and / or the image position of the right-hand gesture after movement to obtain an updated target framing picture.

[0303] In some embodiments, the photographing unit is used to:

[0304] Determining a size change of the framing interest frame and a scaling ratio of the second framing screen according to the image position after the movement of the left-hand gesture and / or the image position after the movement of the right-hand gesture;

[0305] The second framing picture is scaled according to the size change amount and the scaling ratio to obtain an updated target framing picture.

[0306] In some embodiments, the photographing unit is used to:

[0307] After the left hand gesture is released from pinching, when the right hand gesture is pinched and moved, obtaining a position of an image of the right hand gesture after it is moved;

[0308] Determining a picture translation direction and a picture translation amount of the second framing picture according to a position of the image after the movement of the right-hand gesture;

[0309] According to the picture translation direction of the second framing picture and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

[0310] In some embodiments, the photographing unit is used to:

[0311] After the right-hand gesture is released from pinching, when the left-hand gesture is pinched and moved, obtaining a position of an image of the left-hand gesture after it is moved;

[0312] Determining a picture translation direction and a picture translation amount of the second framing picture according to a position of the image of the left-hand gesture after movement;

[0313] According to the picture translation direction of the second framing picture and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

[0314] In some embodiments, the photographing unit is used to:

[0315] Detecting a pinch state of the target gesture based on the operation depth value of the operation key point and the screen depth value;

[0316] When the target gesture is pinched, obtaining the current image position of the target gesture;

[0317] When the target gesture is pinched and moved, obtaining the image position of the target gesture after the movement;

[0318] Determining a picture translation direction and a picture translation amount of the first framing picture according to a current image position of the target gesture and a moved image position of the target gesture;

[0319] According to the picture translation direction and the picture translation amount of the first framing picture, the first framing picture is subjected to translation processing to obtain an updated target scene picture.

[0320] In some embodiments, the reference key points include a first key point and a second key point, and the operation key points include a third key point and a fourth key point.

[0321] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above units can refer to the above gesture key point recognition method embodiment, which will not be repeated here.

[0322] A person skilled in the art will appreciate that all or part of the steps in the above-mentioned gesture key point recognition method may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0323] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute any gesture key point recognition method provided by the embodiment of the present application.

[0324] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0325] In the above-mentioned gesture key point recognition device, computer-readable storage medium, and head-mounted device embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the gesture key point recognition device, computer-readable storage medium, head-mounted device and its corresponding units described above can refer to the description of the gesture key point recognition method in the above embodiment, and will not be repeated here.

[0326] The above is a detailed introduction to a gesture key point recognition method, device, head-mounted device and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A gesture key point recognition method, characterized in that: The method comprises: Obtaining a first viewfinder image of the head mounted device; Perform key point detection based on the first viewfinder image to obtain image two-dimensional coordinates of operation key points of a target gesture in the first viewfinder image and image two-dimensional coordinates of reference key points of the target gesture; Performing projection processing based on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point to obtain second projected three-dimensional coordinates of the operation key point; Obtaining the overall offset of the target gesture; A conversion process is performed according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

2. The method for identifying key points of gestures according to claim 1, characterized in that: The obtaining of the overall offset of the target gesture includes: Obtaining camera intrinsic parameters of the head mounted device; Based on the image two-dimensional coordinates of the operation key point, the second projection three-dimensional coordinates of the operation key point and the camera intrinsic parameter, the overall offset of the target gesture is determined.

3. The method for identifying key points of gestures according to claim 1, characterized in that: The reference key point includes a first key point and a second key point, and the projection processing based on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point to obtain the second projected three-dimensional coordinates of the operation key point includes: Performing weak perspective projection processing on the image two-dimensional coordinates of the operation key point and the image two-dimensional coordinates of the reference key point respectively to obtain the first projection three-dimensional coordinates of the operation key point and the first projection three-dimensional coordinates of the reference key point; Acquire a reference distance between the first key point and the second key point; The second projected three-dimensional coordinates of the operation key point are determined according to the first projected three-dimensional coordinates of the operation key point, the first projected three-dimensional coordinates of the reference key point, and the reference distance.

4. The method for identifying key points of gestures according to claim 1, characterized in that: The method further comprises: Displaying the first framing picture; Based on the camera three-dimensional coordinates of the key operation point, obtaining the operation depth value of the key operation point; According to the operation depth value of the operation key point and the screen depth value of the head mounted device, the first framing screen is updated to obtain an updated target framing screen; A shooting process is performed based on the target framing picture.

5. The method for identifying key points of a gesture according to claim 4, characterized in that: The camera three-dimensional coordinates of the key operation point include a first-axis camera coordinate value of the key operation point, a second-axis camera coordinate value of the key operation point, and a third-axis camera coordinate value of the key operation point; The acquiring the operation depth value of the operation key point based on the camera three-dimensional coordinates of the operation key point includes: The third-axis camera coordinate value of the operation key point is used as the operation depth value of the operation key point.

6. The method for identifying key points of a gesture according to claim 4, characterized in that: The updating of the first framing picture according to the operation depth value of the operation key point and the picture depth value of the head mounted device to obtain an updated target framing picture includes: When the operation depth value of the operation key point increases from small to large until it is equal to the picture depth value, acquiring the image position of the operation key point; According to the image position of the key operation point, obtaining the target object distance of the first framing picture; Focus processing is performed on the first framing picture according to the target object distance to obtain an updated target framing picture.

7. The method for identifying key points of a gesture according to claim 6, characterized in that: The obtaining of the image position of the key operation point comprises: Acquire the camera three-dimensional coordinates of the key operation point, wherein the camera three-dimensional coordinates include the first-axis camera coordinate value of the key operation point, the second-axis camera coordinate value of the key operation point, and the third-axis camera coordinate value of the key operation point; Obtaining camera coordinates of a first corner point of the first framing picture and camera coordinates of a second corner point of the first framing picture; Normalization processing is performed based on the first corner point camera coordinates, the second corner point camera coordinates, the first axis camera coordinate value of the operation key point, and the second axis camera coordinate value of the operation key point to obtain the image position of the operation key point.

8. The method for identifying key points of hand gestures according to claim 4, characterized in that: The target gesture includes a left-hand gesture and a right-hand gesture, and the operation key points include a left-hand key point of the left-hand gesture and a right-hand key point of the right-hand gesture; The updating of the first framing picture according to the operation depth value of the operation key point and the picture depth value of the head mounted device to obtain an updated target framing picture includes: Detecting a pinch state of the left-hand gesture based on the operation depth value of the left-hand key point and the screen depth value; Detecting a pinch state of the right hand gesture based on the operation depth value of the right hand key point and the screen depth value; When the left hand gesture is pinched and the right hand gesture is pinched, a framing interest frame is superimposed on the first framing picture according to the current image position of the left hand gesture and the current image position of the right hand gesture to obtain a second framing picture; The second framing picture is updated according to the image position of the left-hand gesture after movement and / or the image position of the right-hand gesture after movement to obtain an updated target framing picture.

9. The method for identifying key points of a gesture according to claim 8, characterized in that: The updating of the second framing picture according to the image position after the movement of the left hand gesture and / or the image position after the movement of the right hand gesture to obtain an updated target framing picture includes: Determining a size change of the framing interest frame and a scaling ratio of the second framing screen according to the image position after the movement of the left-hand gesture and / or the image position after the movement of the right-hand gesture; The second framing picture is scaled according to the size change amount and the scaling ratio to obtain an updated target framing picture.

10. The method for identifying key points of hand gestures according to claim 8, characterized in that: The updating of the second framing picture according to the image position after the movement of the left hand gesture and / or the image position after the movement of the right hand gesture to obtain an updated target framing picture includes: After the left hand gesture is released from pinching, when the right hand gesture is pinched and moved, obtaining a position of an image of the right hand gesture after it is moved; Determining a picture translation direction and a picture translation amount of the second framing picture according to a position of the image after the movement of the right-hand gesture; According to the picture translation direction of the second framing picture and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

11. The method for identifying key points of hand gestures according to claim 8, characterized in that: The updating of the second framing picture according to the image position after the movement of the left hand gesture and / or the image position after the movement of the right hand gesture to obtain an updated target framing picture includes: After the right-hand gesture is released from pinching, when the left-hand gesture is pinched and moved, obtaining a position of an image of the left-hand gesture after it is moved; Determining a picture translation direction and a picture translation amount of the second framing picture according to a position of the image of the left-hand gesture after movement; According to the picture translation direction of the second framing picture and the picture translation amount of the second framing picture, the second framing picture is subjected to translation processing to obtain an updated target scene picture.

12. The method for identifying key points of hand gestures according to claim 4, characterized in that: The updating of the first framing picture according to the operation depth value of the operation key point and the picture depth value of the head mounted device to obtain an updated target framing picture includes: Detecting a pinch state of the target gesture based on the operation depth value of the operation key point and the screen depth value; When the target gesture is pinched, obtaining the current image position of the target gesture; When the target gesture is pinched and moved, obtaining the image position of the target gesture after the movement; Determining a picture translation direction and a picture translation amount of the first framing picture according to a current image position of the target gesture and a moved image position of the target gesture; According to the picture translation direction and the picture translation amount of the first framing picture, the first framing picture is subjected to translation processing to obtain an updated target scene picture.

13. A gesture key point recognition device, characterized in that: The gesture key point recognition device comprises: A detection unit, used to obtain a first framing image of the head mounted device; The detection unit is used to perform key point detection based on the first viewfinder image to obtain the image two-dimensional coordinates of the operation key points of the target gesture in the first viewfinder image and the image two-dimensional coordinates of the reference key points of the target gesture; A processing unit, configured to perform projection processing based on the two-dimensional image coordinates of the operation key point and the two-dimensional image coordinates of the reference key point to obtain a second projection three-dimensional coordinate of the operation key point; An acquisition unit, used to acquire the overall offset of the target gesture; The conversion unit is used to perform conversion processing according to the overall offset of the target gesture and the second projected three-dimensional coordinates of the operation key point to obtain the camera three-dimensional coordinates of the operation key point.

14. A head mounted device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method for recognizing key points of gestures as claimed in any one of claims 1 to 12 is executed.

15. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the gesture key point recognition method according to any one of claims 1 to 12.

Images