Method for detecting hand joints in glasses equipment, storage medium, electronic equipment and product

Abstract

The invention relates to the technical field of visual inspection, and particularly provides a method for detecting a hand joint in glasses equipment, a storage medium, electronic equipment and a product, and the method can comprise the steps: obtaining the confidence coefficient of a hand region corresponding to a current frame image; under the condition that the confidence coefficient is determined to be greater than a preset threshold value, processing the hand region to obtain a to-be-detected hand region image; and inputting the to-be-detected hand region image into a target hand joint detection model to obtain a hand joint detection result. According to some embodiments of the invention, the accuracy of hand joint detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of visual detection technology, and in particular to a method, storage medium, electronic device and product for detecting hand joints in eyewear equipment. Background Art

[0002] With the continuous development of computer vision technology, visual detection technology has been widely used in detection scenarios. For example, gesture recognition technology combines computer vision, machine learning and image processing technology to detect user gestures. Specifically, user gesture recognition can be achieved by detecting the user's hand joints. When detecting hand joints, the existing technology first roughly determines the heat map of the hand joints, and then performs fitting estimation to obtain the hand joint position. Due to the large number of hand joints, the accuracy of the hand joint position detected by this fitting estimation method is low.

[0003] Therefore, how to provide a technical solution for a method for hand joint detection in an eyewear device with high accuracy has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of some embodiments of the present application is to provide a method, storage medium, electronic device and product for hand joint detection in an eyewear device. The technical solutions of the embodiments of the present application can improve the accuracy of hand joint detection and have high practicality.

[0005] In a first aspect, some embodiments of the present application provide a method for hand joint detection in an eyewear device, comprising: obtaining the confidence of a hand area corresponding to a current frame image; when confirming that the confidence is greater than a preset threshold, processing the hand area to obtain an image of the hand area to be detected; inputting the image of the hand area to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0006] In some embodiments of the present application, when it is determined that the confidence level of the hand area is greater than a preset threshold, the image of the hand area to be detected obtained by processing the hand area is input into the target hand joint detection model to obtain the hand joint detection result. The embodiments of the present application can realize accurate detection of hand joints through the trained target hand joint detection model, and the detection efficiency is high.

[0007] In some embodiments, before inputting the image of the hand area to be detected into the target hand joint detection model, the method also includes: constructing a hand sample data set, the hand sample data set including the hand position area and hand joint standard data in each hand sample of multiple hand samples, wherein the hand joint standard data includes: the three-dimensional coordinates and expected values ​​of each hand joint point; using the hand sample data set to train the initial hand joint detection model to determine the target hand joint detection model.

[0008] Some embodiments of the present application train an initial hand joint detection model through a constructed hand sample data set to obtain a target hand joint detection model, thereby providing model support for subsequent accurate hand joint detection.

[0009] In some embodiments, the three-dimensional coordinates of each hand joint point represent: the two-dimensional position coordinates of each hand joint and the distance from each hand joint to the root area of ​​the middle finger; the expected value represents whether each hand joint point is within the boundary area of ​​each hand sample.

[0010] Some embodiments of the present application characterize the position of each hand joint by three-dimensional coordinates, so as to achieve accurate positioning of the hand joints.

[0011] In some embodiments, constructing a hand sample data set includes: normalizing the acquired hand image data to obtain a processed image; standardizing the hand joints in the processed image to obtain the hand joint standard data; performing a transformation operation on the hand position area in the processed image to obtain the multiple hand samples, wherein the transformation operation includes: offset and / or expansion.

[0012] Some embodiments of the present application can obtain standard and rich sample data by normalizing and standardizing hand image data and transforming the hand position area, thereby providing rich data support for subsequent model training.

[0013] In some embodiments, the training of the initial hand joint detection model includes: a first training stage and a second training stage; wherein, the first training stage uses a hand sample data set that has not been blurred and noise processed for training; and the second training stage uses a hand sample data set that has been blurred and noise processed for training.

[0014] Some embodiments of the present application can improve the accuracy and stability of model training by using different hand sample data sets for training at different training stages.

[0015] In some embodiments, the use of the hand sample data set to train the initial hand joint detection model to determine the target hand joint detection model includes: in the first training stage, the hand position area of ​​each hand sample is converted into a grayscale image of a preset size, and the image is input into the initial hand joint detection model to output hand joint point data; by comparing the hand joint point data with the hand joint standard data, it is determined that the hand joint detection model trained this time meets the preset convergence conditions, and then entering the second training stage; the second training stage is executed on the hand joint detection model trained this time to obtain the target hand joint detection model.

[0016] In some embodiments, the hand area is obtained by the following method: if it is determined that the hand joint detection result of the previous frame image of the current frame image contains no less than a preset number of hand joint information, then the hand area of ​​the current frame image is obtained by using feature point matching; if it is determined that the hand joint detection result of the previous frame image contains less than the preset number of hand joint information, then the hand joint detection model is used to detect the hand area of ​​the current frame image.

[0017] Some embodiments of the present application analyze the hand joint information in the hand joint detection results to determine the detection method for obtaining the hand area of ​​the current frame image, which can effectively reduce computer overhead and improve the efficiency of hand joint detection in consecutive frames.

[0018] In the second aspect, some embodiments of the present application provide a device for hand joint detection in an eyewear device, including: an acquisition module for acquiring the confidence of a hand area corresponding to a current frame image; a processing module for processing the hand area to obtain an image of a hand area to be detected when confirming that the confidence is greater than a preset threshold; and a detection module for inputting the image of the hand area to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0019] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0020] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement a method as described in any embodiment of the first aspect when executing the program.

[0021] In a fifth aspect, some embodiments of the present application provide a computer program product, wherein the computer program product comprises a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1A system diagram of hand joint detection in an eyewear device provided for some embodiments of the present application; Figure 2 A flow chart of a method for obtaining a target hand joint detection model provided in some embodiments of the present application; Figure 3 One of the flow charts of the method for hand joint detection in eyewear devices provided in some embodiments of the present application; Figure 4 A second flow chart of a method for detecting hand joints in an eyewear device provided in some embodiments of the present application; Figure 5 A block diagram of a device for detecting hand joints in an eyewear device provided in some embodiments of the present application; Figure 6 A schematic diagram of an electronic device is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0026] In the related art, with the development of augmented reality AR and virtual reality VR technologies, various types of smart glasses devices continue to emerge. For example, MR glasses (as a specific example of glasses devices) are a kind of smart device that combines AR and VR technologies, which can seamlessly integrate the virtual world with the real world to provide a new visual experience. MR glasses capture objects and actions in the real world through built-in ultrasonic sensors, cameras, LED screens and other technologies, and display virtual images in the glasses, allowing users to see virtual objects in the real environment and interact with them. Currently, gesture recognition is mainly performed through panoramic cameras in glasses devices, but the gesture recognition effect is poor. Therefore, the applicant proposed a method to identify user gestures through hand joint detection. However, the accuracy of hand joint detection in the prior art is low.

[0027] In view of this, some embodiments of the present application provide a method for hand joint detection in an eyewear device, which first determines the hand area from the current frame image, and then uses a trained target hand joint detection model to perform hand joint detection after confirming that the confidence of the hand area is greater than a preset threshold to obtain a hand joint detection result. By analyzing the confidence of the hand area, support can be provided for the accuracy of subsequent hand joint detection, and hand joint detection is only performed on the hand area that meets the confidence requirements, which can reasonably use computer overhead and hardware resources.

[0028] The following is combined with Figure 1 The overall composition structure of the system for hand joint detection in the eyewear device provided by some embodiments of the present application is exemplified.

[0029] like Figure 1 As shown, some embodiments of the present application provide a system for hand joint detection in an eyeglass device, and the system for hand joint detection in the eyeglass device is deployed in the eyeglass device. The system for hand joint detection in the eyeglass device may include: a camera 100 and a detector 200. Among them, the camera 100 can collect video frames or image frames containing the hand of the target object. When the detector 200 receives the current frame image transmitted by the camera 100, the current frame image can be identified to obtain the hand area; then the confidence of the hand area is calculated and analyzed to confirm whether to perform hand joint detection. In the case of confirming to perform hand joint detection, the target hand joint detection model pre-deployed in the detector 200 can detect the hand area image to be detected corresponding to the hand area to obtain the hand joint detection result. Gesture recognition or gesture analysis can be performed subsequently based on the hand joint detection result.

[0030] In some embodiments of the present application, the target hand joint detection model is pre-trained and deployed in the eyewear device; the eyewear device can be AR glasses, VR glasses or MR glasses, etc., and the embodiments of the present application are not specifically limited here.

[0031] It is understandable that in order to achieve accurate detection of hand joints in the current frame image, it is necessary to first obtain the target hand joint detection model. Figure 2 The implementation process of obtaining the target hand joint detection model provided by some embodiments of the present application is exemplified. The following method of obtaining the target hand joint detection model can be executed by the detector 200 or by a server disposed outside the eyewear device, and the embodiments of the present application are not specifically limited here.

[0032] Please see attached Figure 2 , Figure 2 A flow chart of a method for obtaining a target hand joint detection model provided for some embodiments of the present application, the method for obtaining a target hand joint detection model may include: S210, constructing a hand sample data set, wherein the hand sample data set includes a hand position area and hand joint standard data in each hand sample of a plurality of hand samples, wherein the hand joint standard data includes: a three-dimensional coordinate and an expected value of each hand joint point. The three-dimensional coordinate of each hand joint point represents: a two-dimensional position coordinate of each hand joint and a distance from each hand joint to the root area of ​​the middle finger; the expected value represents whether each hand joint point is within the boundary area of ​​each hand sample.

[0033] For example, in some embodiments of the present application, by constructing a hand sample data set containing the 3D coordinates (x, y, z) (as a specific example of three-dimensional coordinates) and valid values ​​(as a specific example of expected values) of each hand joint point (hand joint for short) of the hand, rich training data is provided for subsequent model training. Among them, x, y in (x, y, z) are the position coordinates of the hand joint in the hand sample, and the z value of each hand joint point is mapped to the relative depth (that is, distance) from the root area of ​​the middle finger. In addition, the image of each hand sample is set with an image boundary, and the valid value represents whether each hand joint is within this image boundary. If it is within the image boundary, the valid value is 1, otherwise it is 0. It should be noted that there are 27 hand joints on a normal hand, and 21 hand joints are usually marked in the hand sample. In actual applications, the number of hand joints in the hand sample can be set according to actual needs, so as to train a target hand joint detection model that can identify the corresponding number. The number of hand joints is not specifically limited in the embodiments of the present application.

[0034] In some embodiments of the present application, S210 may include: S211, normalize the acquired hand image data to obtain a processed image.

[0035] For example, in some embodiments of the present application, during the data collection phase, the hand image data may be acquired in at least one of the following three ways: First, obtain from public data sets: for example, collect public image data adapted to the eyewear device scene from data sets such as InterHand, epic-kichen, 3DPose, etc. For example, collect hand image data from the lens perspective, because the distance between the palm and the lens is similar to the camera 100 in the eyewear device.

[0036] Second, generate hand image data: use 3D texture mapping to obtain various forms of 3D hand models. Use the public hand joint motion sequence data to drive the hand 3D model with joints, and project it to a 2D plane according to the camera intrinsic parameters in the eyewear device. The image projected to the 2D plane is fused with a large number of real images that do not contain hands to generate a large amount of hand image data with rich backgrounds, hand shapes, and gestures. The hand image data contains information such as the rectangular position of the hand and the position of the hand joints. It should be noted that the definition of the hand joint position of the above-mentioned 3D hand model is consistent with that mentioned above.

[0037] Third, collect real hand image data: use the target device to collect a large amount of hand image data containing hands, as well as a small amount of pure background samples that do not contain hands; use open source hand detection models (such as yolo8, monado, mediapipe), use multi-model voting to construct hand position samples, and discard samples with inconsistent voting results from multiple models; use the existing open source joint detection model to detect the image in the hand rectangular position (i.e., the hand position), use multi-model voting to construct joint position samples, and discard samples with inconsistent voting results from multiple models, so as to select higher-quality hand image data.

[0038] After obtaining the hand image data in the above manner, the hand image data is subjected to random rotation, perspective transformation and other operations (as a specific example of normalization processing) to obtain richer hand image data and increase the richness of the sample. For the hand joints that are beyond the image boundary after the transformation, the 3D coordinates of the hand joints are set to (0, 0, 0), and the corresponding valid values ​​are set to 0. The hand joints within the image boundary are marked as valid=1, and the 3D coordinates of the hand joints can be determined according to their positions in the image.

[0039] In addition, part of the hand image data may be randomly blurred and noised (for example, Gaussian blur, Gaussian noise, salt and pepper noise, etc.), while part of the data may not be blurred or noised, so as to facilitate the subsequent application of the hand sample data sets to the first training stage and the second training stage, respectively.

[0040] S212, standardizing the hand joints in the processed image to obtain the hand joint standard data.

[0041] For example, in some embodiments of the present application, the definitions of hand joints and the order of hand joints in the hand image data obtained in different ways are uniformly mapped to standard definitions, that is, the form of three-dimensional coordinates mentioned above. For missing hand joints or hand joints beyond the image boundary, the position of the hand joints is set to (0, 0, 0), and the valid value is set to 0.

[0042] S213, performing a transformation operation on the hand position area in the processed image to obtain the multiple hand samples, wherein the transformation operation includes: shifting and / or expanding.

[0043] For example, in some embodiments of the present application, in order to deal with hand position offset, the hand position area in the processed image can also be randomly offset, randomly enlarged, randomly reduced, etc., to improve the robustness and stability of the target hand joint detection model trained subsequently. For example, the offset direction and offset distance are set to implement the offset processing of the hand position area to obtain the offset hand sample; or, the area expansion value of the hand position area is set so that the hand position area can be expanded to the corresponding area area, etc. The embodiments of the present application are not specifically limited here.

[0044] Specifically, when performing random offset, random expansion and other operations on the hand position area, it is first necessary to mark the hand position area in the processed image to obtain a hand position rectangular frame. Use left, top, width, and height to represent the left, top, width, and length of the hand position rectangular frame, respectively. Random offset is performed according to (left, top, width, height), such as adding random noise e, f to left and top, and the range of e, f can be [-10, 10], and then the updated data is obtained, that is, left = left + e, top = top + f. The hand position rectangular frame is randomly expanded, that is, random noise g, h is added to width and height, and the range of g, h can be [-10, 10], and then the updated data is obtained, that is, width = width + g, height = height + h. It can be understood that the random offset, random expansion or random reduction of the hand position rectangular frame can be achieved by the above-mentioned method of adding random noise, so as to obtain a rich hand sample data set for subsequent training. Among them, the value range of random noise can be selected according to actual conditions, and the embodiment of the present application is not specifically limited here.

[0045] In addition, for situations where finger occlusion is difficult to detect, such as fisting, we can maintain as many high-quality hand sample images of this type of gesture as possible in the training hand sample dataset, and generate hand samples with more corresponding joint motion sequences. Among them, the distance and intersection of the finger joint positions can be used to determine whether there is occlusion.

[0046] It should be noted that in order to obtain a hand sample data set with high richness, a variety of image processing methods can be used on the hand image data, such as cropping to a uniform size, etc. The embodiments of the present application are not limited to the above operations.

[0047] S220, using the hand sample data set to train an initial hand joint detection model to determine the target hand joint detection model.

[0048] For example, in some embodiments of the present application, after a rich hand sample data set is constructed through the above embodiments, it can be used to train an initial hand joint detection model to obtain a target hand joint detection model.

[0049] In some embodiments of the present application, the training of the initial hand joint detection model includes: a first training stage and a second training stage; wherein, the first training stage uses a hand sample data set that has not been blurred and noise processed for training; and the second training stage uses a hand sample data set that has been blurred and noise processed for training.

[0050] For example, in some embodiments of the present application, in order to ensure the convergence of the model, in the initial training stage of the model (as a specific example of the first training stage), random blurring and noise settings are not performed, that is, the hand sample data set without blurring and noise processing is used for training. After the model convergence gradually stabilizes, the hand sample data set after blurring and noise processing is added for training. It should be noted that the quality of the model training output results after blurring and noise processing is close to the quality of the output results of the real eyewear device.

[0051] The above process is explained below as an example.

[0052] In some embodiments of the present application, S220 may include: in the first training stage, after converting the hand position area of ​​each hand sample into a grayscale image of a preset size, inputting it into the initial hand joint detection model, and outputting hand joint point data; by comparing the hand joint point data with the hand joint standard data, it is determined that the hand joint detection model trained this time meets the preset convergence conditions, and then entering the second training stage; executing the second training stage on the hand joint detection model trained this time to obtain the target hand joint detection model.

[0053] For example, in some embodiments of the present application, during the model training process, the image of the hand position rectangular frame area (as a specific example of the hand position area) in the hand sample data set is stretched to a grayscale image of 128×128 (as a specific example of the preset size). The grayscale image is then input into the initial hand joint detection model, and the hand joint point data is output, for example, the (x, y, z) values ​​of 21 hand joint points and 21 valid values ​​are output. The output hand joint point data is compared with the hand joint standard data in the hand sample data set to confirm whether the hand joint detection model trained this time meets the preset convergence conditions. If so, the second training stage is entered. After the second training stage is completed, the target hand joint detection model can be obtained. Among them, the above-mentioned preset size can be set flexibly, and the embodiments of the present application are not specifically limited here.

[0054] When entering the second training stage, the training can be based on the hand joint detection model of this training, or after calculating the loss between the hand joint point data and the hand joint standard data, the model of the hand joint detection model of this training is adjusted, and the training is based on the hand joint detection model after the adjustment. The embodiments of the present application are not specifically limited here.

[0055] In addition, to determine whether the hand joint detection model trained this time meets the preset convergence conditions, the hand joint point data and the hand joint standard data can be calculated to obtain the result accuracy. If the result accuracy is higher than the set value, it is considered that the preset convergence conditions are met; or, an error value can be set, and the hand joint point data and the hand joint standard data are calculated to obtain the model error rate. If the model error rate is lower than the set value, it is considered that the preset convergence conditions are met. In addition, the judgment standard for the completion of the second training stage can be that the number of training times has reached the set number, or the accuracy of the model has been equal to or greater than the maximum value, or the error rate of the model has been lower than the minimum value, etc.

[0056] It should be understood that the end conditions of the first training stage and the second training stage can be set according to actual application scenarios, and the embodiments of the present application are not limited thereto.

[0057] The following is combined with Figure 3 The specific process of hand joint detection in the eyewear device performed by the detector 200 provided in some embodiments of the present application is exemplified.

[0058] Please see attached Figure 3 , Figure 3 A flow chart of a method for detecting hand joints in an eyewear device provided for some embodiments of the present application, the method for detecting hand joints in an eyewear device may include: S310, obtaining the confidence of the hand area corresponding to the current frame image.

[0059] For example, in some embodiments of the present application, after receiving the rectangular frame area of ​​the hand position of the current frame image (as a specific example of the hand area), its corresponding confidence is calculated.

[0060] In some embodiments of the present application, S310 may include: if it is determined that the hand joint detection result of the previous frame image of the current frame image contains no less than a preset number of hand joint information, then using feature point matching to obtain the hand area of ​​the current frame image; if it is determined that the hand joint detection result of the previous frame image contains less than the preset number of hand joint information, then using a hand joint detection model to detect the hand area of ​​the current frame image.

[0061] For example, in some embodiments of the present application, the rectangular frame area of ​​the hand position in the current frame image can be detected by the hand detection model, and the acquisition is obtained by matching the feature points with the rectangular frame area of ​​the hand position in the previous frame image. The specific method used is related to the number of hand joints output by the previous frame image (as a specific example of hand joint information).

[0062] If there is a previous frame image for the current frame image, it is necessary to determine whether the number of hand joints detected by the target hand joint detection model for the previous frame image is greater than 10 (as a specific example of the preset number). If so, the rectangular area of ​​the hand position in the current frame image can be detected using feature point matching, otherwise the hand joint detection model is used for detection. If there is no previous frame image for the current frame image, the hand joint detection model is used for detection. In addition, the value of the preset number can be flexibly set according to actual needs, such as 9, 11, etc., and the embodiments of the present application are not limited to this.

[0063] It should be understood that by detecting the hand position using feature point matching under certain conditions, the amount of calculation can be effectively reduced and hardware overhead can be saved.

[0064] S320: When it is confirmed that the confidence level is greater than a preset threshold, the hand region is processed to obtain an image of the hand region to be detected.

[0065] For example, in some embodiments of the present application, if the confidence level is greater than 0.95 (as a specific example of a preset threshold), the camera internal parameters and distortion parameters are used to stretch the rectangular frame area into a 128×128 grayscale image (as a specific example of an image of the hand area to be detected). If the confidence level is not greater than 0.95, it is considered that the detection of the hand position has failed, and the hand joint detection is not performed, and the next frame is directly processed.

[0066] S330, inputting the hand area image to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0067] For example, in some embodiments of the present application, a 128×128 grayscale image is input into a target hand joint detection model trained by the above embodiments to perform joint detection on the target hand joint detection model and obtain a hand joint detection result.

[0068] In one application scenario, if the hand joint data in the hand joint detection result of the current frame image is greater than a preset threshold, the rectangular area of ​​the hand position can be determined for the next frame image by feature point matching, and then the confidence calculation is used to determine whether to perform hand joint detection. That is, the above operation is performed repeatedly until all frame images are detected.

[0069] The following is combined with Figure 4 The specific process of hand joint detection in the eyewear device provided by some embodiments of the present application is exemplified.

[0070] Please see attached Figure 4 , Figure 4 A flow chart of a method for detecting hand joints in an eyewear device is provided for some embodiments of the present application.

[0071] S410, constructing a hand sample dataset.

[0072] S420, training an initial hand joint detection model using a hand sample data set to determine a target hand joint detection model.

[0073] S430, identifying the current frame image to obtain a hand area.

[0074] S440, obtaining the confidence level of the hand region corresponding to the current frame image.

[0075] S450, when it is confirmed that the confidence level is greater than a preset threshold, the hand region is processed to obtain an image of the hand region to be detected.

[0076] S460, inputting the hand area image to be detected into the target hand joint detection model to obtain the hand joint detection result.

[0077] It should be noted that the above embodiment only uses one frame of image as an example to illustrate the implementation process of hand joint detection. In actual applications, there may be a situation where each frame of image in the video frame is continuously detected. Whether it is a single frame of image or continuous tracking detection, the specific implementation process of S410~S460 can refer to the method embodiment provided above. To avoid repetition, the detailed description is appropriately omitted here.

[0078] Please refer to Figure 5 , Figure 5 The following is a block diagram of the components of the device for detecting hand joints in the eyewear device provided by some embodiments of the present application. It should be understood that the device for detecting hand joints in the eyewear device corresponds to the above method embodiment and can execute each step involved in the above method embodiment. The specific functions of the device for detecting hand joints in the eyewear device can be found in the above description. To avoid repetition, the detailed description is appropriately omitted here.

[0079] Figure 5The device for detecting hand joints in the eyeglass device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the device for detecting hand joints in the eyeglass device. The device for detecting hand joints in the eyeglass device includes: an acquisition module 510, used to obtain the confidence of the hand area corresponding to the current frame image; a processing module 520, used to process the hand area to obtain an image of the hand area to be detected when confirming that the confidence is greater than a preset threshold; a detection module 530, used to input the image of the hand area to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0081] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the operations of the method corresponding to any of the above methods provided in the above embodiments.

[0082] Some embodiments of the present application further provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any of the above methods provided in the above embodiments.

[0083] like Figure 6 As shown, some embodiments of the present application provide an electronic device 600, which includes: a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620, wherein the processor 620 can implement a method as described in any of the above embodiments when reading the program from the memory 610 through a bus 630 and executing the program.

[0084] Processor 620 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 620 can be a microprocessor.

[0085] The memory 610 may be used to store instructions executed by the processor 620 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 620 of the disclosed embodiment may be used to execute instructions in the memory 610 to implement the method shown above. The memory 610 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0086] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0087] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for detecting hand joints in an eyewear device, characterized in that: include: Get the confidence of the hand area corresponding to the current frame image; When it is confirmed that the confidence level is greater than a preset threshold, the hand region is processed to obtain an image of the hand region to be detected; Inputting the hand area image to be detected into a target hand joint detection model to obtain a hand joint detection result, wherein the target hand joint detection model is obtained by training with a hand sample data set; The three-dimensional coordinates of each hand joint point in the hand sample training data set represent the two-dimensional position coordinates of each hand joint and the distance from each hand joint to the root area of ​​the middle finger, and the expected value represents whether each hand joint point is within the boundary area of ​​each hand sample.

2. The method according to claim 1, characterized in that Before inputting the hand area image to be detected into the target hand joint detection model, the method further includes: Constructing a hand sample data set, wherein the hand sample data set includes a hand position area and hand joint standard data in each hand sample of a plurality of hand samples, wherein the hand joint standard data includes: a three-dimensional coordinate and an expected value of each hand joint point; The initial hand joint detection model is trained using the hand sample data set to determine the target hand joint detection model.

3. The method according to claim 2, characterized in that The step of constructing a hand sample data set includes: Normalizing the acquired hand image data to obtain a processed image; Standardizing the hand joints in the processed image to obtain standard data of the hand joints; Performing a transformation operation on the hand position area in the processed image to obtain the multiple hand samples, wherein the transformation operation includes: shifting and / or expanding.

4. The method according to claim 2, characterized in that The training of the initial hand joint detection model includes: a first training stage and a second training stage; wherein the first training stage uses a hand sample data set that has not been subjected to blurring and noise processing for training; and the second training stage uses a hand sample data set that has been subjected to the blurring and noise processing for training.

5. The method according to claim 4, characterized in that The using the hand sample data set to train the initial hand joint detection model to determine the target hand joint detection model includes: In the first training stage, the hand position area of ​​each hand sample is converted into a grayscale image of a preset size, and then input into an initial hand joint detection model to output hand joint point data; By comparing the hand joint point data with the hand joint standard data, after determining that the hand joint detection model trained this time meets the preset convergence condition, entering the second training stage; The second training phase is performed on the hand joint detection model trained this time to obtain the target hand joint detection model.

6. The method according to any one of claims 1 to 2 and 5, characterized in that: The hand region is obtained by the following method: If it is determined that the hand joint detection result of the previous frame image of the current frame image contains no less than a preset amount of hand joint information, then using feature point matching to obtain the hand region of the current frame image; If it is determined that the hand joint detection result of the previous frame image contains less than the preset amount of hand joint information, the hand joint detection model is used to detect the hand area of ​​the current frame image.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the computer program executes the method according to any one of claims 1 to 6 when being run by the processor.

9. A computer program product, characterized in that The computer program product comprises a computer program, wherein the computer program executes the method according to any one of claims 1 to 6 when executed by a processor.

Images