Gesture recognition key point marking method for unmanned loader, medium and program product

Abstract

The invention discloses an unmanned loader gesture recognition key point marking method, a medium and a program product, and the method comprises the steps: respectively obtaining a key point video with a label and a key point video without a label, and pasting a preset color and shape label at a to-be-tracked key point part on a to-be-marked object in the key point video with the label; the key point part to be tracked on the object to be marked in the key point video without the label is not pasted with any label; objects to be labeled in the key point video with the label and the key point video without the label move in the same mode; tracking positions and names of key points in the key point video with the label by using a tracking algorithm, and generating a labeling file of the object with the label; and migrating the labeled file of the object with the label to the key point video without the label, and generating a key point labeled file of the object without the label. According to the method, the problem of jitter during key point labeling can be avoided, and the key point data set manufacturing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of gesture recognition, and in particular relates to a method, medium and program product for marking key points of gesture recognition for an unmanned loader. Background Art

[0002] The equipment manufacturing industry is moving towards high-end and intelligent development. Under this trend, intelligent and unmanned construction machinery has become the development trend and general consensus of the industry. In the process of unmanned loaders, key point estimation is a relatively important link and has been applied to gesture recognition and truck compartment corner point recognition. However, the existing methods for preparing key point datasets for gesture recognition have the following problems:

[0003] (1) Existing gesture recognition key point datasets generally use labeling software such as Labelme to directly label the location information and name information of each key point on each image. The labeling process is cumbersome and the labeling efficiency is low.

[0004] (2) Each annotator has a different annotation style, and annotators are prone to fatigue after a long period of annotation, resulting in problems such as jitter in the position of the annotation key points and missing key points, which in turn leads to poor results in subsequent network training. Summary of the invention

[0005] In view of the above problems, the present invention proposes a method, medium and program product for marking key points of gesture recognition for unmanned loader, which can avoid the jitter problem during key point marking and improve the efficiency and accuracy of key point data set production.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, the present invention provides a method for marking key points of gesture recognition, comprising:

[0008] Obtaining a labeled key point video and an unlabeled key point video respectively, wherein the key point parts to be tracked on the object to be labeled in the labeled key point video are labeled with preset colors and shapes; the key point parts to be tracked on the object to be labeled in the unlabeled key point video are not labeled; the object to be labeled in the labeled key point video and the unlabeled key point video moves in the same way;

[0009] Use tracking algorithms to track the locations and names of key points in the labeled key point video and generate annotation files of labeled objects;

[0010] The annotation file of the labeled object is migrated to the key point video without labels, and the key point annotation file of the unlabeled object is generated, so as to realize the key point tracking on the unlabeled object to be labeled.

[0011] In combination with the first aspect, optionally, the preset color and shape label is a white circular label; and the method for generating a labeling file of the labeled object includes:

[0012] Extract all video frames of the video with labeled key points;

[0013] Automatically filter out the white area in the first video frame by color space conversion and threshold segmentation method;

[0014] Use the Hough circle transform algorithm to detect circles in the white area and output the coordinates of the center position of the circle;

[0015] Use annotation software to label the key points of the detected white circular area;

[0016] The RGB images of all video frames, as well as the center position coordinates and key point names corresponding to the first video frame, are input into the DINO-Tracker tracking algorithm to obtain the key point tracking results of each frame in the remaining video frames in the labeled key point video, and saved in the JSON annotation file format to generate an annotation file with labeled objects.

[0017] In combination with the first aspect, optionally, the step of automatically screening out the white area in the first video frame by using the color space conversion and threshold segmentation method includes:

[0018] Convert the first video frame from the RGB color space to the HSV color space using a color space conversion;

[0019] Find white areas in HSV color space.

[0020] In combination with the first aspect, optionally, converting the color space of the first video frame from RGB to HSV by using image color space conversion includes:

[0021] Traverse the RGB color space of the first video frame and calculate the maximum value and minimum value ;

[0022] According to the maximum value and minimum value , calculate hue , Saturation and brightness ;

[0023] Among them, the maximum value and minimum value The calculation formula is:

[0024] ;

[0025] Hue , Saturation and brightness The calculation formula is:

[0026] ;

[0027] in, are the red, green and blue light intensities, is the maximum value and minimum value The difference between

[0028] The method of searching for a white area in the HSV color space includes:

[0029] Find the white area according to the following threshold setting:

[0030] ;

[0031] in, and Represent the minimum and maximum hue values, respectively. and Represent the minimum saturation value and the maximum saturation value respectively. and Represent the minimum and maximum brightness values, respectively. and Indicates the brightness threshold and saturation threshold of the white area in the HSV color space.

[0032] In combination with the first aspect, optionally, the detecting a circle in the white area using the Hough circle transform algorithm and outputting the coordinates of the center position of the circle includes:

[0033] The Hough circle transform algorithm converts points in the image space into the parameter space and accumulates votes in the parameter space to determine the center position and radius length.

[0034] In combination with the first aspect, optionally, the DINO-Tracker tracking algorithm performs the following steps on the received data:

[0035] The RGB images of all video frames are input into the pre-trained DINOv2-ViT model for image feature extraction. The center position coordinates in the first video frame are used as the initial key point position. The key point names and key point positions are stored as a list and sorted.

[0036] based on Frame and The image features of the frame calculate the cosine similarity of each key point. When the cosine similarity is greater than or equal to the set threshold, Frame and The image features of the frame are input into the Delta-DINO model to calculate the residual of the matching key points. When the residual is less than or equal to the displacement threshold, the key point position in the next frame is predicted, and the key point state is marked as visible in the next frame, and the key point position is updated to the list, otherwise the key point state is marked as invisible in the next frame. When the key point is invisible, the key point name and key point position are deleted from the list; until the key point tracking results of each frame in the remaining video frames in the labeled key point video are obtained and saved as a list.

[0037] In combination with the first aspect, optionally, the method for generating a labeling file of the labeled object includes:

[0038] Each video frame is encoded into a string in Base64 format and saved as an imageData attribute, wherein the imageData attribute includes the video frame number, key point name and circle center position coordinates, wherein the key point name and circle center position coordinates are saved in a dictionary format.

[0039] In combination with the first aspect, optionally, migrating the labeled object annotation file to an unlabeled key point video includes:

[0040] Encode each video frame of the unlabeled keypoint video into a Base64 format string;

[0041] The Base64 format string of each video frame of the unlabeled key point video is replaced by the imageData attribute of the corresponding video frame of the labeled key point video to generate a key point annotation file for the unlabeled object, thereby realizing the migration of the key points of the labeled object to the key points of the unlabeled object.

[0042] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for labeling key points of gesture recognition as described in any one of the first aspects.

[0043] In a third aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the gesture recognition key point labeling method described in any one of the first aspects.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention first labels the key point positions, uses a visual algorithm to automatically track the positions and names of the labeled key points, generates a labeling file for the labeled objects, and migrates the JSON labeling file of the labeled objects to the unlabeled objects, avoiding the jitter problem during key point labeling and improving the efficiency and accuracy of key point data set production.

[0046] The present invention automatically tracks the position and name of the key point of the starting frame and automatically labels each subsequent frame of video, thereby simplifying the labeling process and improving the labeling efficiency.

[0047] The present invention uses a graphical processing method to automatically detect and track key points, thereby avoiding the problem of position jitter of key points and missing key points caused by different annotation styles or fatigue of annotators. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor, among which:

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 It is a schematic diagram of an initial frame of a video with labeled key points of the present invention;

[0051] Figure 3 This is a schematic diagram of an initial frame of a video without labeled key points of the present invention;

[0052] Figure 4 It is a schematic diagram of key point detection results of the present invention;

[0053] Figure 5 is a schematic diagram of key point tracking results of the present invention;

[0054] Figure 6 This is a schematic diagram of the video annotation result without label key points of the present invention;

[0055] FIG7 (a) is a schematic diagram of the training result without using the present invention;

[0056] FIG. 7( b ) is a schematic diagram of the training result using the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] Example 1

[0060] An embodiment of the present invention provides a method for marking key points of gesture recognition, comprising the following steps:

[0061] (1) Obtaining a labeled key point video and an unlabeled key point video respectively, wherein the key point parts to be tracked on the object to be labeled in the labeled key point video are labeled with preset colors and shapes; the key point parts to be tracked on the object to be labeled in the unlabeled key point video are not labeled; and the object to be labeled in the labeled key point video and the unlabeled key point video move in the same manner;

[0062] (2) Use a tracking algorithm to track the positions and names of key points in the labeled key point video and generate a label file of the labeled objects;

[0063] (3) The annotation file of the labeled object is migrated to the key point video without labels, and the key point annotation file of the unlabeled object is generated, so as to realize the key point tracking on the unlabeled object to be labeled.

[0064] In a specific implementation of the embodiment of the present invention, the preset color and shape label is a white circular label; the method for generating a labeling file of the labeled object includes:

[0065] Extract the first video frame of the video with labeled key points;

[0066] Automatically filter out the white area in the first video frame by color space conversion and threshold segmentation method;

[0067] Use the Hough circle transform algorithm to detect circles in the white area and output the coordinates of the center position of the circle;

[0068] Use labeling software (such as Labelme software) to label the key points of the detected white circular area;

[0069] The RGB image, center position coordinates and key point names of the first video frame are input into the DINO-Tracker tracking algorithm to obtain the key point tracking results of each frame in the remaining video frames in the labeled key point video, and saved in the JSON annotation file format to generate an annotation file with labeled objects.

[0070] In a specific implementation of the embodiment of the present invention, the method of automatically screening out the white area in the first video frame by using the color space conversion and threshold segmentation method includes:

[0071] Convert the first video frame from the RGB color space to the HSV color space using a color space conversion;

[0072] Find white areas in HSV color space.

[0073] In a specific implementation of the embodiment of the present invention, converting the color space of the first video frame from RGB to HSV by using image color space conversion includes:

[0074] Traverse the RGB color space of the first video frame and calculate the maximum value and minimum value ;

[0075] According to the maximum value and minimum value , calculate hue , Saturation and brightness ;

[0076] Among them, the maximum value and minimum value The calculation formula is:

[0077] ;

[0078] Hue , Saturation and brightness The calculation formula is:

[0079] ;

[0080] in, are the red, green and blue light intensities, is the maximum value and minimum value The difference between

[0081] The method of searching for a white area in the HSV color space includes:

[0082] Find the white area according to the following threshold setting:

[0083] ;

[0084] in, and Represent the minimum and maximum hue values, respectively. and Represent the minimum saturation value and the maximum saturation value respectively. and Represent the minimum and maximum brightness values, respectively. and Represents the brightness threshold and saturation threshold of the white area in the HSV color space, which are set to 225 and 15 respectively in the experiment.

[0085] In a specific implementation of the embodiment of the present invention, the method of using the Hough circle transform algorithm to detect a circle in the white area and outputting the coordinates of the center position of the circle includes:

[0086] The Hough circle transform algorithm converts points in the image space into the parameter space and accumulates votes in the parameter space to determine the center position and radius length of the circle.

[0087] In a specific implementation of the embodiment of the present invention, the DINO-Tracker tracking algorithm performs the following steps on the received data:

[0088] The RGB images of all video frames are input into the pre-trained DINOv2-ViT model for image feature extraction. The center position coordinates in the first video frame are used as the initial key point position. The key point names and key point positions are stored as a list and sorted.

[0089] based on Frame and The image features of the frame calculate the cosine similarity of each key point. When the cosine similarity is greater than or equal to the set threshold (such as 0.8), Frame and The image features of the frame are input into the Delta-DINO model to calculate the residual of the matching key points. When the residual is less than or equal to the displacement threshold, the key point position in the next frame is predicted, and the key point status is marked as visible in the next frame, and the key point position is updated to the list. Otherwise, the key point status is marked as invisible in the next frame. When the key point is invisible, the key point name and key point position are deleted from the list; until the key point tracking results of each frame in the remaining video frames in the labeled key point video are obtained and saved as a list. In the process of predicting the key point position in the next frame, the following formula is used:

[0090] ;

[0091] ;

[0092] ;

[0093] In the formula, and Respectively represent the pre-trained DINOv2-ViT model in Frame and The semantic features extracted from the frame, for Frame and Frame The similarity calculation results of key points are: represents the cosine similarity function, For the predicted Frame The location of the key points. It is the Delta-DINO model, which is used to output the residual of the features extracted by DINOv2-ViT. is the displacement threshold, Reason The predicted trajectory is is the true trajectory, For displacement loss, the network predicts the trajectory and the true trajectory Displacement loss optimization Delta-DINO model between.

[0094] In a specific implementation of the embodiment of the present invention, the method for generating a labeling file of a labeled object includes:

[0095] Each video frame is encoded into a string in Base64 format and saved as an imageData attribute, wherein the imageData attribute includes the video frame number, key point name and circle center position coordinates, wherein the key point name and circle center position coordinates are saved in a dictionary format.

[0096] In a specific implementation of the embodiment of the present invention, migrating the labeled object annotation file to the key point video without labels includes:

[0097] Encode each video frame of the unlabeled keypoint video into a Base64 format string;

[0098] The Base64 format string of each video frame of the unlabeled key point video is replaced by the imageData attribute of the corresponding video frame of the labeled key point video to generate a key point annotation file for the unlabeled object, thereby realizing the migration of the key points of the labeled object to the key points of the unlabeled object.

[0099] The method for marking key points for gesture recognition in an embodiment of the present invention is described in detail below in conjunction with a specific implementation.

[0100] like Figure 1 As shown, a method for marking key points of gesture recognition for an unmanned loader includes the following steps:

[0101] Step 1: Labeling, that is, attaching white circular labels to the key points to be tracked on the object to be labeled;

[0102] like Figure 2 As shown, white circular labels with a diameter of 0.8 cm are pasted at key points.

[0103] Step 2: Move the object and collect the key point video with labels;

[0104] Use the Kinect binocular vision camera on the unmanned loader to collect video. The video resolution is set to 640*480, and the video frame rate is set to 30FPS. Move the object for which key point data is to be collected to the center of the video, and collect labeled key point videos. The collection time of each video is 5 seconds.

[0105] Step 3: Remove the label, move the object in the same way, and collect key point videos without labels;

[0106] like Figure 3 As shown in the figure, when collecting videos of key points without labels, the shape of the object and its position in the video are kept unchanged, while the speed, acceleration, and trajectory of the object remain unchanged, the frame rate of the video collected by the camera remains unchanged, and the length of the collected video remains unchanged. To increase the diversity of collected samples, external factors such as light source and background can be changed.

[0107] Step 4: Use the tracking algorithm to track the positions and names of key points in the labeled key point video, and generate a key point annotation file (i.e., JSON annotation file) for the labeled object.

[0108] First, extract the first frame of the video with labeled key points, automatically filter out the white area in the video frame through color space conversion and threshold segmentation method, use the Hough circle transform algorithm to detect the circle in the white area and output the coordinates of the center position of the circle, and finally use Labelme software to annotate the key point names of the detected white circular area. The first frame of the video with labeled key points is shown in the figure below. Figure 4 shown.

[0109] Among them, the color space conversion converts the color space of the video frame from RGB to the color space HSV (Hue, Saturation, Value) which is more suitable for color segmentation. The HSV color space can better separate color information, making color segmentation easier to achieve. The formula for calculating Hue, Saturation, Value is as follows:

[0110] Traverse the RGB color space to calculate the maximum and minimum values:

[0111]

[0112] According to the maximum value and minimum value Calculate the hue, saturation and brightness:

[0113]

[0114] Find the white area in the HSV color space and set the following threshold:

[0115]

[0116] in, and Represent the minimum and maximum hue values, respectively. and Represent the minimum saturation value and the maximum saturation value respectively. and Represent the minimum and maximum brightness values, respectively. and Represents the brightness threshold and saturation threshold of the white area in the HSV color space, which are set to 225 and 15 respectively in the experiment.

[0117] The Hough circle transform algorithm converts points in the image space into the parameter space (i.e., the three-dimensional space of the center position (x, y) and radius r), and accumulates votes in the parameter space to determine the center position and radius length. In the specific implementation, the cv2.HoughCircles() function in the OpenCV library is used to implement the Hough circle transform to detect circles in the white area and output the coordinates of the center position.

[0118] Then, the RGB image of the first frame of the labeled key point video and the center coordinates of the white circular label are input into the DINO-Tracker tracking algorithm together with the name, and the key point tracking results of each frame in the remaining video frames are obtained and saved in the JSON annotation file format. The key point tracking results of the remaining video frames are as follows: Figure 5 shown.

[0119] The DINO-Tracker tracking algorithm performs the following steps on the received data:

[0120] The RGB images of all video frames are input into the pre-trained DINOv2-ViT model for image feature extraction. The center position coordinates in the first video frame are used as the initial key point position. The key point names and key point positions are stored as a list and sorted.

[0121] based on Frame and The image features of the frame calculate the cosine similarity of each key point. When the cosine similarity is greater than or equal to the set threshold (such as 0.8), Frame and The image features of the frame are input into the Delta-DINO model to calculate the residual of the matching key points. When the residual is less than or equal to the displacement threshold, the key point position in the next frame is predicted, and the key point status is marked as visible in the next frame, and the key point position is updated to the list. Otherwise, the key point status is marked as invisible in the next frame. When the key point is invisible, the key point name and key point position are deleted from the list; until the key point tracking results of each frame in the remaining video frames in the labeled key point video are obtained and saved as a list. In the process of predicting the key point position in the next frame, the following formula is used:

[0122] ;

[0123] ;

[0124] ;

[0125] In the formula, and Respectively represent the pre-trained DINOv2-ViT model in Frame and The semantic features extracted from the frame, for Frame and Frame The similarity calculation results of key points are: represents the cosine similarity function, For the predicted Frame The location of the key points. It is the Delta-DINO model, which is used to output the residual of the features extracted by DINOv2-ViT. is the displacement threshold, Reason The predicted trajectory is is the true trajectory, For displacement loss, the network predicts the trajectory and the true trajectory Displacement loss optimization Delta-DINO model between.

[0126] Each video frame is encoded as a string in Base64 format and saved as the imageData attribute; the key point name and coordinates are saved in dictionary format. When a key point in a video frame is out of the field of view or is blocked and invisible, the key point is not recorded.

[0127] Step 5: Migrate the JSON annotation file of the labeled object to the unlabeled object, generate the key point annotation file of the unlabeled object, and realize the key point tracking on the unlabeled object.

[0128] After tracking all video frames, the unlabeled object video is extracted. Each video frame is also encoded as a Base64 format string. The imageData attribute of the JSON annotation file in the corresponding labeled video frame is used to replace the Base64 format string to generate the key point annotation file of the unlabeled object, so as to realize the migration of the key points of the labeled object to the key points of the unlabeled object. The migrated unlabeled object key point video frame is as follows: Figure 6 shown.

[0129] The training results of using the present invention and not using the present invention are shown in FIG7 , where FIG7 (a) is the training result of the labeled data set of the present invention, and FIG7 (b) is the training result of the manually labeled data set. The same training hyperparameters are used, and the statistical training results are shown in Table 1. Wherein, train / box_loss represents the training box loss; train / obj_loss represents the training target loss; train / cls_loss represents the training classification loss; metrics / precision represents the precision rate; and metrics / recall represents the recall rate. val / box_loss represents the verification box loss; vall / obj loss represents the verification target loss; val / cls loss represents the verification classification loss; metrics / mAP 0.5 represents the average precision mean IoU of 0~50%; metrics / mAP 0.5:0.95 represents the average precision mean IoU of 50~95%.

[0130] As can be seen from Table 1, the present invention can effectively improve the training results of the network. The network has improved by 3.36% in precision (Precision), 14.59% in recall (Recall), 11.65% when the mean precision mean (MAP) IoU is 0-50%, and 11.82% when the IoU is 50-95%. It can also be observed from the table that after using the present invention, when the network uses a larger intersection-over-union ratio on the mean precision, the performance of the network is improved more. This shows that the network can better identify and locate targets under stricter detection standards, with higher accuracy and robustness, which further verifies that the present invention improves the accuracy of key point annotation while improving the efficiency of key point data set production.

[0131] Table 1 Network training results

[0132] Performance Indicators Precision / % Recall / % MAP_50 / % MAP_95 / % The present invention 72.31 88.61 92.64 75.87 Manual annotation 68.95 74.02 80.99 64.05 Performance Improvements +3.36 +14.59 +11.65 +11.82

[0133] The test platform and experimental environment of the present invention are: Windows 10 Professional operating system, Intel(R) Xeon(R) Gold 6248R CPU, 128GB memory, NVIDIA RTX 3090 GPU, Windows 10 operating system, Pytorch 1.10.2, CUDA 11.4, and Python 3.8.12.

[0134] Example 2

[0135] In an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for labeling key points of gesture recognition described in any one of Embodiment 1 is implemented.

[0136] Example 3

[0137] In an embodiment of the present invention, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the method for labeling key points of gesture recognition described in any one of the embodiment 1 is implemented.

[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

[0143] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for marking key points of gesture recognition, characterized in that: include: Obtaining a labeled key point video and an unlabeled key point video respectively, wherein the key point parts to be tracked on the object to be labeled in the labeled key point video are labeled with preset colors and shapes; the key point parts to be tracked on the object to be labeled in the unlabeled key point video are not labeled; the object to be labeled in the labeled key point video and the unlabeled key point video moves in the same way; Use tracking algorithms to track the locations and names of key points in the labeled key point video and generate annotation files of labeled objects; The annotation file of the labeled object is migrated to the key point video without labels, and the key point annotation file of the unlabeled object is generated, so as to realize the key point tracking on the unlabeled object to be labeled.

2. A method for marking key points of gesture recognition according to claim 1, characterized in that: The preset color and shape label is a white circular label; The method for generating a labeling file of a labeled object includes: Extract all video frames of the video with labeled key points; Automatically filter out the white area in the first video frame by color space conversion and threshold segmentation method; Use the Hough circle transform algorithm to detect circles in the white area and output the coordinates of the center position of the circle; Use annotation software to label the key points of the detected white circular area; The RGB images of all video frames, as well as the center position coordinates and key point names corresponding to the first video frame, are input into the DINO-Tracker tracking algorithm to obtain the key point tracking results of each frame in the remaining video frames in the labeled key point video, and saved in the JSON annotation file format to generate an annotation file with labeled objects.

3. A method for marking key points of gesture recognition according to claim 2, characterized in that: The method of automatically screening out the white area in the first video frame by using the color space conversion and threshold segmentation method includes: Convert the first video frame from the RGB color space to the HSV color space using a color space conversion; Find white areas in HSV color space.

4. The method for marking key points of gesture recognition according to claim 3, characterized in that: The method of converting the color space of the first video frame from RGB to HSV by using the color space conversion includes: Traverse the RGB color space of the first video frame and calculate the maximum value and minimum value ; According to the maximum value and minimum value , calculate hue , Saturation and brightness ; Among them, the maximum value and minimum value The calculation formula is: ； Hue , Saturation and brightness The calculation formula is: ； in, are the red, green and blue light intensities, is the maximum value and minimum value The difference between The method of searching for a white area in the HSV color space includes: Find the white area according to the following threshold setting: ； in, and Represent the minimum and maximum hue values, respectively. and Represent the minimum saturation value and the maximum saturation value respectively. and Represent the minimum and maximum brightness values, respectively. and Indicates the brightness threshold and saturation threshold of the white area in the HSV color space.

5. The method for marking key points of gesture recognition according to claim 3, characterized in that: The method of using the Hough circle transform algorithm to detect a circle in the white area and outputting the coordinates of the center position of the circle includes: The Hough circle transform algorithm converts points in the image space into the parameter space and accumulates votes in the parameter space to determine the center position and radius length of the circle.

6. The method for marking key points of gesture recognition according to claim 2, characterized in that: The DINO-Tracker tracking algorithm performs the following steps on the received data: The RGB images of all video frames are input into the pre-trained DINOv2-ViT model for image feature extraction. The center position coordinates in the first video frame are used as the initial key point position. The key point names and key point positions are stored as a list and sorted. based on Frame and The image features of the frame calculate the cosine similarity of each key point. When the cosine similarity is greater than or equal to the set threshold, Frame and The image features of the frame are input into the Delta-DINO model to calculate the residual of the matching key points. When the residual is less than or equal to the displacement threshold, the key point position in the next frame is predicted, and the key point state is marked as visible in the next frame, and the key point position is updated to the list, otherwise the key point state is marked as invisible in the next frame. When the key point is invisible, the key point name and key point position are deleted from the list; until the key point tracking results of each frame in the remaining video frames in the labeled key point video are obtained and saved as a list.

7. The method for marking key points of gesture recognition according to claim 2, characterized in that: The method for generating a labeling file of a labeled object includes: Each video frame is encoded into a string in Base64 format and saved as an imageData attribute, wherein the imageData attribute includes the video frame number, key point name and circle center position coordinates, wherein the key point name and circle center position coordinates are saved in a dictionary format.

8. The method for marking key points of gesture recognition according to claim 8, characterized in that: The step of migrating the labeled object annotation file to the key point video without labels includes: Encode each video frame of the unlabeled keypoint video into a Base64 format string; The Base64 format string of each video frame of the unlabeled key point video is replaced by the imageData attribute of the corresponding video frame of the labeled key point video to generate a key point annotation file for the unlabeled object, thereby realizing the migration of the key points of the labeled object to the key points of the unlabeled object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for labeling key points of gesture recognition described in any one of claims 1 to 8 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for labeling key points of gesture recognition described in any one of claims 1 to 8 is implemented.

Images