Method for detecting hand joint in eyewear device, storage medium, electronic device and product

Abstract

The application relates to the technical field of visual detection, and particularly provides a method for detecting hand joints in a glasses device, a storage medium, an electronic device and a product. The method can comprise the following steps: acquiring a confidence of a hand region corresponding to a current frame image; in the case that it is confirmed that the confidence is greater than a preset threshold, processing the hand region to obtain a hand region image to be detected; and inputting the hand region image to be detected into a target hand joint detection model to acquire a hand joint detection result. Some embodiments of the application can improve the accuracy of hand joint detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual detection, in particular to a method for detecting hand joints in eyewear equipment, a storage medium, an electronic device and a product. BACKGROUND

[0002] With the continuous development of computer vision technology, visual detection technology is widely applied to detection scenes. For example, gesture recognition technology combines computer vision, machine learning and image processing technology to realize detection of user gestures. Specifically, user gestures can be recognized by detecting hand joints of the user. In the prior art, a hotspot map of a hand joint is first determined roughly, and then fitting estimation is performed to obtain the position of the hand joint. Since there are many hand joints, the accuracy of the position of the hand joint detected by the fitting estimation is low.

[0003] Therefore, how to provide a technical solution of a method for detecting hand joints in eyewear equipment with high accuracy becomes a technical problem to be solved urgently. SUMMARY

[0004] Some embodiments of the present application aim to provide a method for detecting hand joints in eyewear equipment, a storage medium, an electronic device and a product. The technical solution of the embodiments of the present application can improve the accuracy of hand joint detection and has high practicability.

[0005] In a first aspect, some embodiments of the present application provide a method for detecting hand joints in eyewear equipment, comprising: obtaining a confidence of a hand region corresponding to a current frame image; in a case where it is confirmed that the confidence is greater than a preset threshold, processing the hand region to obtain a hand region image to be detected; and inputting the hand region image to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0006] Some embodiments of the present application input the hand region image to be detected obtained by processing the hand region in a case where the confidence of the hand region is greater than the preset threshold into the target hand joint detection model to obtain the hand joint detection result. The embodiments of the present application can realize accurate detection of hand joints by using the trained target hand joint detection model, and have high detection efficiency.

[0007] In some embodiments, before the hand region image to be detected is input into the target hand joint detection model, the method further comprises: constructing a hand sample data set, wherein the hand sample data set includes a hand position region and hand joint standard data in each hand sample of a plurality of hand samples, and the hand joint standard data includes three-dimensional coordinates and an expected value of each hand joint node; and training an initial hand joint detection model by using the hand sample data set to determine the target hand joint detection model.

[0008] Some embodiments of the present application train an initial hand joint detection model to obtain a target hand joint detection model through a constructed hand sample data set, thereby providing model support for subsequent accurate hand joint detection.

[0009] In some embodiments, the three-dimensional coordinates of each hand joint represent the two-dimensional position coordinates of each hand joint and the distance from each hand joint to the middle finger proximal region; and the expected value represents whether each hand joint is within the boundary region of each hand sample.

[0010] Some embodiments of the present application represent the position of each hand joint by three-dimensional coordinates, which can achieve accurate positioning of the hand joint.

[0011] In some embodiments, the hand sample data set is constructed by: performing normalization processing on the obtained hand image data to obtain a processed image; performing standardization on the hand joints in the processed image to obtain hand joint standard data; and performing transformation operation on the hand position region in the processed image to obtain the plurality of hand samples, wherein the transformation operation includes shifting and / or expanding.

[0012] Some embodiments of the present application can obtain standard and rich sample data by performing normalization processing, standardization processing on hand image data, and transformation operation on the hand position region, thereby providing rich data support for subsequent model training.

[0013] In some embodiments, the training of the initial hand joint detection model includes a first training stage and a second training stage; the first training stage uses a hand sample data set that has not undergone blur processing and noise processing; and the second training stage uses a hand sample data set that has undergone the blur processing and the noise processing.

[0014] Some embodiments of the present application can improve the accuracy and stability of model training by using different hand sample data sets in different training stages.

[0015] In some embodiments, the training of the initial hand joint detection model using the hand sample data set to determine the target hand joint detection model includes: in the first training stage, converting the hand position region of each hand sample into a preset size of a grayscale image and inputting the grayscale image into an initial hand joint detection model to output hand joint node data; after determining that the hand joint detection model of this training meets a preset convergence condition by comparing the hand joint node data with the hand joint standard data, entering the second training stage; and performing the second training stage on the hand joint detection model of this training to obtain the target hand joint detection model.

[0016] In some embodiments, the hand region is obtained by: if it is determined that the hand joint detection result of the previous frame image of the current frame image contains hand joint information no less than a preset number, then the hand region of the current frame image is obtained by using feature point matching; and if it is determined that the hand joint detection result of the previous frame image contains hand joint information less than the preset number, then the hand region of the current frame image is detected by using a hand joint detection model.

[0017] Some embodiments of the present application can effectively reduce computer overhead and improve hand joint detection efficiency in consecutive frames by analyzing hand joint information in a hand joint detection result to determine a detection method for obtaining a hand region of a current frame image.

[0018] In a second aspect, some embodiments of the present application provide a device for hand joint detection in a glasses device, comprising: an obtaining module configured to obtain a confidence of a hand region corresponding to a current frame image; a processing module configured to, in a case where it is confirmed that the confidence is greater than a preset threshold, process the hand region to obtain a hand region image to be detected; and a detection module configured to input the hand region image to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0019] In a third aspect, some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, can implement the method according to any one of the embodiments of the first aspect.

[0020] In a fourth aspect, some embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method according to any one of the embodiments of the first aspect.

[0021] In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, can implement the method according to any one of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0023] Figure 1System diagram of hand joint detection in eyewear device provided for some embodiments of the present application;

[0024] Figure 2 Method flow diagram of acquiring target hand joint detection model provided for some embodiments of the present application;

[0025] Figure 3 Method flow diagram one of hand joint detection in eyewear device provided for some embodiments of the present application;

[0026] Figure 4 Method flow diagram two of hand joint detection in eyewear device provided for some embodiments of the present application;

[0027] Figure 5 Device composition block diagram of hand joint detection in eyewear device provided for some embodiments of the present application;

[0028] Figure 6 Electronic device schematic diagram provided for some embodiments of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

[0030] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms “first”, “second”, etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0031] In related art, with the development of augmented reality (AR) and virtual reality (VR) technologies, various types of smart eyewear devices have emerged. For example, MR glasses (as a specific example of eyewear devices) are a smart device that combines AR and VR technologies, which can seamlessly integrate virtual worlds with real worlds, providing a new visual experience. MR glasses capture real-world objects and actions through built-in ultrasonic sensors, cameras, and LED screens, and display virtual images within the glasses, allowing users to see virtual objects in a real environment and interact with them. Currently, hand gesture recognition in eyewear devices is mainly performed by panoramic cameras, but this hand gesture recognition effect is poor. Therefore, the applicant proposes a way to recognize user gestures through hand joint detection. However, the accuracy of hand joint detection in existing technology is low.

[0032] In view of this, some embodiments of the present application provide a method for hand joint detection in a glasses device, which first determines a hand region from a current frame image, and then, when it is confirmed that the confidence of the hand region is greater than a preset threshold, uses a trained target hand joint detection model to perform hand joint detection to obtain a hand joint detection result. Through analysis of the confidence of the hand region, the accuracy of subsequent hand joint detection can be supported, and only the hand regions that meet the confidence requirement are subjected to hand joint detection, so that computer overheads can be reasonably used and hardware resources can be reasonably utilized.

[0033] The overall structure of a system for hand joint detection in a glasses device provided by some embodiments of the present application will be described below with reference to the accompanying Figure 1 The overall structure of a system for hand joint detection in a glasses device provided by some embodiments of the present application will be described below with reference to the accompanying

[0034] As Figure 1 shown, some embodiments of the present application provide a system for hand joint detection in a glasses device, which is deployed in a glasses device. The system for hand joint detection in a glasses device can include a camera 100 and a detector 200. The camera 100 can collect a video frame or an image frame containing a hand of a target object. When the detector 200 receives a current frame image from the camera 100, it can identify and obtain a hand region from the current frame image, and then calculate and analyze the confidence of the hand region to determine whether to perform hand joint detection. When it is determined to perform hand joint detection, a target hand joint detection model pre-deployed in the detector 200 can detect a to-be-detected hand region image corresponding to the hand region to obtain a hand joint detection result. Subsequently, gesture recognition or gesture analysis can be performed according to the hand joint detection result.

[0035] In some embodiments of the present application, the target hand joint detection model is pre-trained and deployed in the glasses device; the glasses device can be an AR glasses, a VR glasses, or an MR glasses, which is not specifically limited in the embodiments of the present application.

[0036] It can be understood that, in order to achieve accurate hand joint detection on the current frame image, a target hand joint detection model needs to be obtained first. The implementation process of obtaining the target hand joint detection model provided by some embodiments of the present application will be described below with reference to the accompanying Figure 2 The implementation process of obtaining the target hand joint detection model provided by some embodiments of the present application will be described below with reference to the accompanying

[0037] Please refer to the accompanying Figure 2 , Figure 2 A method flowchart for obtaining a target hand joint detection model provided by some embodiments of the present application can include:

[0038] S210, construct a hand sample data set, wherein the hand sample data set includes a hand position region and hand joint standard data in each hand sample of a plurality of hand samples, and wherein the hand joint standard data includes three-dimensional coordinates of each hand joint and an expected value. The three-dimensional coordinates of each hand joint represent two-dimensional position coordinates of each hand joint and a distance from each hand joint to a middle finger knuckle region, and the expected value represents whether each hand joint is within a boundary region of each hand sample.

[0039] For example, in some embodiments of the present application, a hand sample data set containing 3D coordinates (x, y, z) (as a specific example of three-dimensional coordinates) and valid values (as a specific example of expected values) of each hand joint of a hand is constructed to provide rich training data for subsequent model training. Among them, x and y in (x, y, z) are the position coordinates of the hand joint in the hand sample, and the z value of each hand joint is mapped to the relative depth (i.e., distance) to the middle finger knuckle region. In addition, the image of each hand sample is provided with an image boundary, and the valid value represents whether each hand joint is within the image boundary. If it is within the image boundary, the valid value is 1, otherwise it is 0. It should be noted that a normal hand has 27 hand joints, and usually 21 hand joints are labeled in a hand sample. In actual application, the number of hand joints in the hand sample can be set according to actual needs, so as to train a target hand joint detection model that can recognize the corresponding number of hand joints. The number of hand joints is not specifically limited in the embodiments of the present application.

[0040] In some embodiments of the present application, S210 can include:

[0041] S211, performing normalization processing on the obtained hand image data to obtain a processed image.

[0042] For example, in some embodiments of the present application, in the data collection stage, the hand image data can be obtained by at least one of the following three ways:

[0043] First, obtain from a public data set: for example, collect public image data suitable for the scene of the glasses device from the InterHand, epic-kitchen, 3DPose, etc. data sets. For example, collect hand image data from the lens perspective, because the distance between the palm and the lens is similar to the camera 100 in the glasses device.

[0044] Second, generate hand image data: use 3D texture map to get various shapes of hand 3D model. Use the public hand joint motion sequence data to drive the hand 3D model, and project it to the 2D plane according to the camera internal parameter in the glasses device, and fuse the image projected to the 2D plane with a large number of real images not containing hands, to generate a large number of hand image data with rich background, hand shape and gesture. The hand image data contains information such as the rectangular position of the hand and the position of the hand joint. It should be noted that the definition of the hand joint position of the hand 3D model in the above is consistent with the above-mentioned definition.

[0045] Third, collect real hand image data: collect a large amount of hand image data containing hands and a small amount of pure background samples not containing hands with the target device; use the open source hand detection model (such as yolo8, monado, mediapipe) to construct the hand position sample by using the multi-model voting method, and discard the samples whose voting results of multiple models are inconsistent; for the image in the hand rectangular position (i.e. hand position), use the existing open source joint detection model for detection, and use the multi-model voting method to construct the joint position sample, and discard the samples whose voting results of multiple models are inconsistent, so as to select hand image data with higher quality.

[0046] After obtaining the hand image data in the above-mentioned manner, random rotation, perspective transformation and other operations (as a specific example of normalization processing) are performed on the hand image data to obtain more rich hand image data and increase the richness of the samples. For the hand joints that exceed the image boundary after transformation, the 3D coordinates of the hand joints are set to (0, 0, 0), and the corresponding valid value is set to 0. The hand joints within the image boundary are marked as valid=1, and the 3D coordinates of the hand joints are determined according to their positions in the image.

[0047] In addition, part of the hand image data can be randomly blurred and noise processed (for example, Gaussian blur, Gaussian noise, salt and pepper noise, etc.), and part of the data is not blurred and noise processed, so as to apply the hand sample data set to the subsequent first training stage and second training stage respectively in the subsequent process.

[0048] S212, standardize the hand joints in the processed image to obtain the hand joint standard data.

[0049] For example, in some embodiments of the present application, the definition of the hand joints in the hand image data obtained by using different methods and the definition of the hand joint sequence are uniformly mapped to a standard definition, i.e. the form of the three-dimensional coordinates mentioned above. For the missing hand joints or the hand joints exceeding the image boundary, the position of the hand joint is set to (0, 0, 0) and the valid value is set to 0.

[0050] S213, performing a transformation operation on the hand position region in the processed image to obtain the plurality of hand samples, wherein the transformation operation comprises: shifting and / or expanding.

[0051] For example, in some embodiments of the present application, in order to cope with the hand position shift, random shift, random expansion, random reduction, etc. operations can also be performed on the hand position region in the processed image to improve the robustness and stability of the target hand joint detection model in subsequent training. For example, the shift direction and shift distance are set to realize the shift processing of the hand position region to obtain the shifted hand sample; or the area expansion value of the hand position region is set so that the hand position region can be expanded to the corresponding area region, etc. The embodiments of the present application are not limited here.

[0052] Specifically, when performing random shift, random expansion, etc. operations on the hand position region, first, the hand position rectangle frame is obtained by labeling the hand position region in the processed image. left, top, width, and height are used to represent the left, top, width, and length of the hand position rectangle frame. According to (left, top, width, height), random shift is performed, such as adding random noise e, f to left and top, and the range of e, f can be [-10, 10], and then the updated data is obtained, i.e. left = left + e, top = top + f. Random expansion is performed on the hand position rectangle frame, i.e. random noise g, h is added to width and height, and the range of g, h can be [-10, 10], and then the updated data is obtained, i.e. width = width + g, height = height + h. It can be understood that the random shift, random expansion or random reduction, etc. of the hand position rectangle frame can be realized by the above-mentioned way of adding random noise, to obtain a rich hand sample dataset for subsequent training. The range of the random noise can be selected according to the actual situation, and the embodiments of the present application are not limited here.

[0053] In addition, for the detection difficulty situation of clenched fist and other finger occlusion, the proportion of as many high-quality hand sample pictures as possible of this kind of gesture in the training hand sample dataset can be maintained, and more corresponding joint motion sequences are used to generate hand samples. Among them, whether there is occlusion can be judged by using the distance and intersection of finger joint positions.

[0054] It should be noted that in order to obtain a hand sample dataset with high richness, a plurality of image processing methods can be used for hand image data, such as cropping to a uniform size, etc. The embodiments of the present application are not limited to the above operations.

[0055] S220, training the initial hand joint detection model using the hand sample dataset to determine the target hand joint detection model.

[0056] For example, in some embodiments of this application, after a rich hand sample dataset is constructed through the above embodiments, it can be used to train the initial hand joint detection model to obtain the target hand joint detection model.

[0057] In some embodiments of this application, the training of the initial hand joint detection model includes: a first training phase and a second training phase; wherein, the first training phase is trained using a hand sample dataset that has not undergone blurring and noise processing; and the second training phase is trained using a hand sample dataset that has undergone blurring and noise processing.

[0058] For example, in some embodiments of this application, to ensure model convergence, random blurring and noise addition are not performed during the initial training phase (as a specific example of the first training phase); that is, training is performed using a hand sample dataset without blurring and noise addition. The blurred and noise-added hand sample dataset is then added for training after the model has gradually stabilized. It should be noted that the quality of the model training output after blurring and noise addition is close to the quality of the output from a real glasses device.

[0059] The above process is illustrated below by example.

[0060] In some embodiments of this application, S220 may include: in the first training stage, converting the hand position region of each hand sample into a grayscale image of a preset size, inputting it into an initial hand joint detection model, and outputting hand joint point data; by comparing the hand joint point data with the standard hand joint data, determining that the hand joint detection model trained this time meets the preset convergence condition, and then entering the second training stage; performing the second training stage on the hand joint detection model trained this time to obtain the target hand joint detection model.

[0061] For example, in some embodiments of this application, during model training, the image of the rectangular region of the hand position in the hand sample dataset (as a specific example of the hand position region) is stretched to a grayscale image of 128×128 (as a specific example of the preset size). Then, the grayscale image is input into the initial hand joint detection model, outputting hand joint data, for example, outputting the (x, y, z) values ​​of 21 hand joints and 21 valid values. The output hand joint data is compared with the standard hand joint data in the hand sample dataset to confirm whether the trained hand joint detection model meets the preset convergence condition. If it does, the second training stage begins. After the second training stage is completed, the target hand joint detection model can be obtained. The preset size can be flexibly set, and this embodiment of the application does not impose specific limitations on it.

[0062] When entering the second training stage, the hand joint detection model of the current training can be used as the basis for training, or the hand joint detection model of the current training can be trained based on the hand joint detection model after the hand joint data and the hand joint standard data are used to calculate the loss and the hand joint detection model is adjusted.

[0063] In addition, whether the hand joint detection model of the current training meets the preset convergence condition can be determined by calculating the hand joint data and the hand joint standard data to obtain the result accuracy. If the result accuracy is higher than the set value, it is considered that the preset convergence condition is met. Alternatively, an error value can be set, and the model error rate can be obtained by calculating the hand joint data and the hand joint standard data. If the model error rate is lower than the set value, it is considered that the preset convergence condition is met. In addition, the judgment standard for completing the second training stage can be that the number of training times reaches the set number, or the accuracy of the model is equal to or greater than the maximum value, or the error rate of the model is lower than the minimum value, etc.

[0064] It should be understood that the end conditions of the first training stage and the second training stage can be set according to the actual application scene, and the embodiments of the present application are not limited thereto.

[0065] The specific process of the hand joint detection in the glasses device performed by the detector 200 will be described below with reference to the accompanying drawings. Figure 3 The specific process of the hand joint detection in the glasses device performed by the detector 200 will be described below with reference to the accompanying drawings.

[0066] Please refer to the accompanying drawings Figure 3 , Figure 3 A method flowchart for hand joint detection in a glasses device is provided for some embodiments of the present application. The method for hand joint detection in the glasses device can include:

[0067] S310, obtaining the confidence of the hand region corresponding to the current frame image.

[0068] For example, in some embodiments of the present application, after receiving the rectangular frame region of the hand position of the current frame image (as a specific example of the hand region), the corresponding confidence is calculated.

[0069] In some embodiments of the present application, S310 can include: if it is determined that the hand joint detection result of the previous frame image of the current frame image contains no less than a preset number of hand joint information, the hand region of the current frame image is obtained by using feature point matching; if it is determined that the hand joint detection result of the previous frame image contains less than the preset number of hand joint information, the hand region of the current frame image is detected by using the hand joint detection model.

[0070] For example, in some embodiments of the present application, the rectangular frame region of the hand position in the current frame image can be detected by the hand detection model, and the acquisition is obtained by feature point matching with the rectangular frame region of the hand position in the previous frame image. Which specific way is used is related to the number of hand joints output by the previous frame image (as a specific example of hand joint information).

[0071] If the current frame image exists the previous frame image, it is needed to judge whether the number of hand joints detected by the target hand joint detection model in the previous frame image is greater than 10 (as a specific example of the preset number), if yes, the rectangular region of the hand position in the current frame image can be detected by the feature point matching mode, otherwise the hand joint detection model is used for detection. If the current frame image does not exist the previous frame image, the hand joint detection model is used for detection. In addition, the value of the preset number can be flexibly set according to actual needs, for example, 9, 11, etc., and the embodiments of the present application are not limited thereto.

[0072] It should be understood that by using the feature point matching mode under certain conditions to detect the hand position, the computational complexity can be effectively reduced, and the hardware overhead can be saved.

[0073] S320, in the case where the confidence is greater than a preset threshold, processing the hand region to obtain a to-be-detected hand region image.

[0074] For example, in some embodiments of the present application, if the confidence is greater than 0.95 (as a specific example of the preset threshold), the rectangular frame region is stretched into a 128x128 gray image (as a specific example of the to-be-detected hand region image) by using the camera intrinsic parameters and distortion parameters. If the confidence is not greater than 0.95, it is considered that the detection of the hand position fails, and the hand joint detection is not performed, and the next frame is directly processed.

[0075] S330, inputting the to-be-detected hand region image into the target hand joint detection model to obtain a hand joint detection result.

[0076] For example, in some embodiments of the present application, the 128x128 gray image is input into the target hand joint detection model trained by the above-mentioned embodiments to perform joint detection, and a hand joint detection result is obtained.

[0077] In an application scenario, if the hand joint data in the hand joint detection result of the current frame image is greater than a preset threshold, the rectangular region of the hand position in the next frame image can be determined by using the feature point matching mode, and then it is determined by the confidence calculation whether to perform the hand joint detection. That is, the above-mentioned operation is repeatedly executed until all frame images are detected.

[0078] The following will be described in combination with the drawingsFigure 4 An example illustrates a specific process of hand joint detection in the eyeglasses device provided by some embodiments of the present application.

[0079] Please refer to the accompanying drawings Figure 4 , Figure 4 A method flowchart of hand joint detection in the eyeglasses device provided by some embodiments of the present application is shown.

[0080] S410, a hand sample data set is constructed.

[0081] S420, the initial hand joint detection model is trained by using the hand sample data set to determine a target hand joint detection model.

[0082] S430, the current frame image is recognized to obtain a hand region.

[0083] S440, the confidence of the hand region corresponding to the current frame image is obtained.

[0084] S450, the hand region is processed to obtain a hand region image to be detected when the confidence is greater than a preset threshold.

[0085] S460, the hand region image to be detected is input into the target hand joint detection model to obtain a hand joint detection result.

[0086] It should be noted that the above embodiments only take one frame of image as an example to illustrate the implementation process of hand joint detection, and in actual application, there may be a case of continuous detection of each frame of image in a video frame. Whether it is a frame of image or continuous tracking detection, the specific implementation process of S410-S460 can refer to the method embodiments provided above, and to avoid repetition, the detailed description is appropriately omitted here.

[0087] Please refer to the accompanying drawings Figure 5 , Figure 5 A composition block diagram of the device for hand joint detection in the eyeglasses device provided by some embodiments of the present application is shown. It should be understood that the device for hand joint detection in the eyeglasses device corresponds to the above method embodiments, and can perform each step involved in the above method embodiments. The specific functions of the device for hand joint detection in the eyeglasses device can be referred to the description in the above, and to avoid repetition, the detailed description is appropriately omitted here.

[0088] Figure 5The device for hand joint detection in the eyewear device includes at least one software function module that can be stored in a memory in a form of software or firmware or solidified in the device for hand joint detection in the eyewear device, and the device for hand joint detection in the eyewear device includes: an acquisition module 510 configured to acquire a confidence of a hand region corresponding to a current frame image; a processing module 520 configured to, when it is confirmed that the confidence is greater than a preset threshold, process the hand region to obtain a hand region image to be detected; and a detection module 530 configured to input the hand region image to be detected into a target hand joint detection model to obtain a hand joint detection result.

[0089] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the foregoing method, and will not be described in more detail here.

[0090] Some embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the operations of the method corresponding to any of the embodiments of the foregoing method provided by the foregoing embodiments.

[0091] Some embodiments of the present application also provide a computer program product, which includes a computer program, wherein the computer program is executed by a processor to implement the operations of the method corresponding to any of the embodiments of the foregoing method provided by the foregoing embodiments.

[0092] As shown in Figure 6 Some embodiments of the present application provide an electronic device 600, which includes a memory 610, a processor 620, and a computer program stored in the memory 610 and executable on the processor 620, wherein the processor 620 reads the program from the memory 610 through a bus 630 and executes the program to implement the method of any of the foregoing embodiments.

[0093] The processor 620 can process digital signals and can include various computing structures. For example, a complex instruction set computer structure, a reduced instruction set computer structure, or a structure implementing a combination of multiple instruction sets. In some examples, the processor 620 can be a microprocessor.

[0094] The memory 610 can be used to store instructions executed by the processor 620 or data related to the instructions execution process. The instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 620 of the embodiments of the present disclosure can be configured to execute the instructions in the memory 610 to implement the methods shown in the above. The memory 610 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory well known to those skilled in the art.

[0095] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0096] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0097] It should be noted that, in the present document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

Claims

1. A method for detecting hand joints in an eyeglass device, characterized in that, The method comprises the following steps: obtaining a confidence of a hand region corresponding to a current frame image; if it is confirmed that the confidence is greater than a preset threshold, processing the hand region to obtain a hand region image to be detected; inputting the hand region image to be detected into a target hand joint detection model to obtain a hand joint detection result, wherein the target hand joint detection model is obtained by training a hand sample data set, the three-dimensional coordinates of each hand joint node in the hand sample training data set represent the two-dimensional position coordinates of each hand joint and the distance from each hand joint to the middle finger root region, and the expected value represents whether each hand joint node is in the boundary region of each hand sample; wherein the hand region is obtained by the following method: if it is determined that the hand joint detection result of the previous frame image of the current frame image contains no less than a preset number of hand joint information, the hand region of the current frame image is obtained by using feature point matching; if it is determined that the hand joint detection result of the previous frame image contains less than the preset number of hand joint information, the hand region of the current frame image is detected by using a hand joint detection model.

2. The method of claim 1, wherein, Before the hand region image to be detected is input into the target hand joint detection model, the method further comprises the following steps: constructing a hand sample data set, wherein the hand sample data set comprises a hand position region and hand joint standard data in each hand sample of a plurality of hand samples, and the hand joint standard data comprises three-dimensional coordinates of each hand joint node and an expected value; training an initial hand joint detection model using the hand sample data set to determine the target hand joint detection model.

3. The method of claim 2, wherein, The construction of the hand sample data set comprises the following steps: performing normalization processing on the obtained hand image data to obtain a processed image; standardizing the hand joints in the processed image to obtain the hand joint standard data; performing a transformation operation on the hand position region in the processed image to obtain the plurality of hand samples, wherein the transformation operation comprises shifting and / or expanding.

4. The method of claim 2, wherein, The training of the initial hand joint detection model comprises a first training stage and a second training stage; wherein the first training stage uses a hand sample data set that has not been subjected to blur processing and noise processing for training; and the second training stage uses a hand sample data set that has been subjected to the blur processing and the noise processing for training.

5. The method of claim 4, wherein, The training of the initial hand joint detection model using the hand sample data set to determine the target hand joint detection model comprises the following steps: in the first training stage, after converting the hand position region of each hand sample into a grayscale image of a preset size, inputting the grayscale image into the initial hand joint detection model to output hand joint node data; after determining that the hand joint detection model of this training meets a preset convergence condition by comparing the hand joint node data with the hand joint standard data, entering the second training stage; performing the second training stage on the hand joint detection model of this training to obtain the target hand joint detection model.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by the processor to execute the method in any one of claims 1-5.

7. An electronic device, comprising: A computer program product comprises a computer program, and the computer program is run by the processor to execute the method in any one of claims 1-5.

8. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is run by the processor to execute the method in any one of claims 1-5.

Images