Hand image labeling method and device, wearable device and storage medium

Abstract

The embodiment of the invention provides a hand image labeling method, a hand image labeling device, a wearable device and a storage medium. The method comprises the following steps: acquiring a hand image of a wearer through a fisheye camera on the wearable equipment, and performing segmentation processing operation on the hand image to obtain a hand segmented image; mapping the hand segmentation image to a preset background to obtain a mapped hand segmentation image; and performing fisheye distortion operation on the mapped hand segmentation image to obtain a fisheye-distorted hand segmentation image, and determining a target gesture frame based on the fisheye-distorted hand segmentation image. The embodiment of the invention aims to reduce the cost of acquiring the hand image and marking the hand image, and ensure the stability of the marking result.

Description

Technical Field

[0001] The present application relates to the field of intelligent wearable technology, and in particular to a hand image annotation method, a hand image annotation device, a wearable device and a computer-readable storage medium. Background Art

[0002] As a convenient and natural way of human-computer interaction, gesture recognition has been widely used in smart wearable fields such as AR (Augmented Reality) glasses and VR (Virtual Reality) glasses.

[0003] At present, gesture recognition methods based on deep learning have achieved remarkable results. However, they require a large amount of labeled gesture data for training. Therefore, the training process before gesture recognition is not only time-consuming and expensive, but also not conducive to obtaining stable and relatively accurate gesture recognition results. In addition, traditional gesture recognition methods usually rely on RGB (Red, Green, Blue) cameras for the collection of gesture images, which have a small field of view and high power consumption. Therefore, they are not suitable for the collection of gesture images by mobile terminals such as smart glasses. Summary of the invention

[0004] The present application provides a hand image annotation method, a hand image annotation device, a wearable device and a computer-readable storage medium, aiming to reduce the hand image acquisition cost and the hand image annotation cost, and ensure the stability of the annotation results.

[0005] To achieve the above object, the present application provides a hand image annotation method, the method comprising:

[0006] The fisheye camera on the wearable device collects a hand image of the wearer, and performs a segmentation processing operation on the hand image to obtain a hand segmentation image;

[0007] Mapping the hand segmentation image to a preset background to obtain a mapped hand segmentation image;

[0008] A fisheye distortion operation is performed on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image, and a corresponding target gesture frame is determined based on the fisheye distorted hand segmentation image.

[0009] To achieve the above objectives, the present application also provides a hand image annotation device, comprising:

[0010] An acquisition module, wherein the acquisition module is used to acquire a hand image of the wearer through a fisheye camera on the wearable device, and perform a segmentation processing operation on the hand image to obtain a hand segmentation image;

[0011] A mapping module, wherein the mapping module is used to map the hand segmentation image to a preset background to obtain a mapped hand segmentation image;

[0012] A determination module is used to perform a fisheye distortion operation on the mapped hand segmentation image to obtain a hand segmentation image after fisheye distortion, and determine a corresponding target gesture frame based on the hand segmentation image after fisheye distortion.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a wearable device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of any one of the hand image annotation methods provided in the embodiments of the present application when executing the computer program.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the hand image annotation method described in any one of the embodiments of the present application.

[0015] The hand image annotation method, hand image annotation device, wearable device and computer-readable storage medium disclosed in the embodiment of the present application can collect the hand image of the wearer through the fisheye camera on the wearable device, and perform segmentation processing operation on the hand image to obtain a hand segmentation image. Then, the hand segmentation image can be mapped to a preset background to obtain a mapped hand segmentation image. Further, the mapped hand segmentation image can be subjected to fisheye distortion operation to obtain a hand segmentation image after fisheye distortion, and the corresponding target gesture frame is determined based on the hand segmentation image after fisheye distortion. The present application can collect hand images through a fisheye camera with low power consumption and a large viewing angle range, and reduce the acquisition cost of hand images on the basis of improving the acquisition efficiency of hand images. Furthermore, after obtaining the hand image through the fisheye camera, the hand image is sequentially subjected to segmentation processing operation and background mapping operation. Since the hand segmentation image is mapped to a variety of preset backgrounds, the richness of the hand segmentation image is improved, thereby improving the generalization ability of the target gesture frame. Furthermore, the above method directly determines the corresponding target gesture frame based on the hand segmentation image after fisheye distortion, that is, the target gesture frame can be obtained by only one annotation. Therefore, on the basis of improving the generation efficiency of the target gesture frame, the stability of the target gesture frame is also improved, and the annotation cost of the gesture frame is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 It is a flowchart of a hand image annotation method provided in an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of a process of obtaining a hand segmentation image provided by an embodiment of the present application;

[0019] Figure 3 is another schematic diagram of a process for obtaining a hand segmentation image provided by an embodiment of the present application;

[0020] Figure 4 is another schematic diagram of a scene for obtaining a hand segmentation image provided by an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a scene of a hand segmentation image provided by an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of a process of mapping a hand segmentation image to a preset background provided by an embodiment of the present application;

[0023] Figure 7 It is a schematic diagram of a process of obtaining a hand segmentation image after fisheye distortion provided by an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of an isometric model provided in an embodiment of the present application;

[0025] Fig. 9 is a schematic block diagram of a hand image annotation device provided in an embodiment of the present application;

[0026] Fig.10 It is a schematic block diagram of a wearable device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0029] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0031] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for labeling a hand image provided in an embodiment of the present application. Figure 1 As shown, the hand image labeling method includes steps S11 to S14.

[0032] Step S11: The fisheye camera on the wearable device is used to capture the wearer's hand image, and the hand image is segmented to obtain a hand segmentation image.

[0033] It should be noted that a wearable device is an intelligent device that integrates computer technology, communication technology and sensor technology, and can be worn on the wearer's body, such as wrist, head, glasses, shoes, etc. Therefore, the wearable device can collect the wearer's data in real time and process the data to interact with the wearer, and then provide the wearer with corresponding functions or services.

[0034] The embodiment of the present application is described by taking a wearable device as smart glasses as an example, wherein the smart glasses include a fisheye camera for capturing a hand image of the wearer, and then identifying the hand image of the wearer to achieve interaction with the wearer.

[0035] Compared with the related art that usually collects the wearer's hand image based on RGB (Red, Green, Blue) camera, the present application uses a fisheye camera with the advantages of a larger field of view and lower power consumption, which is beneficial to reduce the cost of collecting hand images and thus improve the efficiency of collecting the wearer's hand images.

[0036] Specifically, the fisheye camera can be used to capture the wearer's hand image. The captured hand image includes the wearer's gesture image and the background excluding the gesture image. Therefore, the captured hand image can be segmented to obtain a hand segmentation image containing only the gesture image.

[0037] It should be noted that the present application does not limit the segmentation processing operation, such as threshold segmentation, edge segmentation, and cutout operation. It can be understood that by performing segmentation processing operations on the hand image to obtain a hand segmentation image containing only the gesture image, the gesture image can be separated from the original background, preventing the problem of weak generalization ability of gesture recognition due to a single background.

[0038] In an embodiment of the present application, a fisheye camera can be used to capture the wearer's hand image, and the hand image can be segmented and processed to obtain a hand segmentation image. Since the fisheye camera has the advantages of low power consumption and a large viewing angle, it can reduce power consumption and gesture acquisition costs in the gesture acquisition process while improving the efficiency of gesture acquisition. In addition, by segmenting the hand image to obtain a hand segmentation image containing only the gesture image, the gesture image can be separated from the original background, preventing the problem of weak generalization ability of gesture recognition due to a single background.

[0039] Step S12: Map the hand segmentation image to a preset background to obtain a mapped hand segmentation image.

[0040] It should be noted that the present application does not limit the preset background. For example, it can be an indoor scene (such as an office, living room, kitchen, etc.), an outdoor scene (such as a park, street, beach, etc.), or a virtual reality scene (simulating a virtual reality environment through the display module of a smart wearable device).

[0041] It can be understood that by mapping the hand segmentation image to a diverse background, the richness of the hand segmentation image can be improved, thereby improving the generalization ability in the gesture recognition process.

[0042] It should be noted that the present application does not limit the method of mapping the hand segmentation image to the preset background. For example, the hand segmentation image can be synthesized into the preset background by means of transparency synthesis or fusion algorithm.

[0043] Optionally, the hand segmentation image can also be subjected to data enhancement operations, such as flipping the image, rotating the image, enlarging the image, or reducing the image, so as to ensure data diversity, quickly synthesize a large amount of data, reduce the subsequent gesture frame annotation cost, and thereby improve the stability of gesture recognition results.

[0044] In the embodiment of the present application, the hand segmentation image can be mapped to a preset background to obtain a mapped hand segmentation image, thereby improving the generalization ability in the gesture recognition process and the stability of the gesture recognition result.

[0045] Step S13: performing a fisheye distortion operation on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image, and determining a corresponding target gesture frame based on the fisheye distorted hand segmentation image.

[0046] It should be noted that the fisheye distortion operation refers to processing an image or video to simulate the image or video distortion effect caused by a gray-white fisheye lens.

[0047] Furthermore, the present application does not limit the method of fisheye distortion operation, such as including geometric transformation method, image synthesis method, etc. The present application takes the fisheye distortion operation including geometric transformation method as an example for explanation.

[0048] The geometric transformation method includes pole projection and tangent projection. Specifically, based on the preset requirements or expected effects, the isometric model can be determined by selecting a suitable focal length or field of view, and then the isometric model is used to transform the coordinates of each pixel in the image to simulate the distortion effect of the fisheye lens. Therefore, the present application can obtain the hand segmentation image after fisheye distortion based on the geometric transformation method.

[0049] In addition, a gesture box refers to a bounding box used to mark and locate the position of a gesture in an image or video in the field of computer vision and human-computer interaction. A gesture box is usually a rectangular box that surrounds the area where the hand or gesture is located so that the gesture can be further analyzed and recognized.

[0050] Therefore, after obtaining the hand segmentation image after fisheye distortion, the corresponding target gesture frame can be determined based on the hand segmentation image after fisheye distortion. There can be multiple target gesture frames. Thus, several target gesture frames can be determined as annotation data, and the annotation data can be used to train the pre-trained model to obtain a gesture recognition model.

[0051] In the embodiment of the present application, the mapped hand segmentation image can be subjected to fisheye distortion to obtain a fisheye distorted hand segmentation image, and then the target gesture frame can be obtained. Since the target gesture frame is also a fisheye distorted image, it can better adapt to changes and noise in the real world and has certain stability and generalization ability.

[0052] Optionally, after determining the corresponding target gesture frame based on the hand segmentation image after fisheye distortion, the method further includes: training a pre-trained model using the target gesture frame to obtain a gesture recognition model.

[0053] It should be noted that the present application does not limit the pre-trained model, and includes, for example, a convolutional neural network, a recurrent neural network, etc. It can be understood that the gesture recognition model is a model trained based on the pre-trained model.

[0054] Specifically, in order to obtain a gesture recognition model, a label can be assigned to the gesture action in each target gesture frame. For example, if the gesture in the target gesture frame is "open palm", the corresponding label can be "palm_open". Gesture labels should include various possible gesture actions, such as "clenching fist", "pointing", "thumb up", etc. Each label represents a gesture category so that the gesture recognition model can learn to recognize different gestures. Furthermore, by associating each target gesture frame with the corresponding gesture label, a data set can be created, in which the data set contains the target gesture frame and its corresponding label to ensure that a variety of different gestures are included in the data set. By inputting the data set into the pre-training model for training, the pre-model can learn to distinguish different gesture actions during the training process, thereby training a gesture recognition model. The gesture recognition model can predict the label of the gesture action based on the input gesture image, thereby realizing automatic recognition and classification of gestures.

[0055] Optionally, after determining the corresponding target gesture frame based on the hand segmentation image after fisheye distortion, the method further includes: performing a feature extraction operation on the target gesture frame to obtain a corresponding feature vector; and analyzing the feature vector to obtain a gesture recognition result.

[0056] Specifically, the target gesture frame may be subjected to a feature extraction operation to convert the image information in the target gesture frame into a corresponding feature vector. Furthermore, the feature vector may be analyzed by a gesture recognition model, and a gesture recognition result may be obtained according to the output of the gesture recognition model.

[0057] In the embodiment of the present application, the target gesture frame can be used to train the pre-trained model to obtain a gesture recognition model. In this way, gesture recognition can be implemented based on the gesture recognition model.

[0058] The hand image annotation method disclosed in the embodiment of the present application can collect the hand image of the wearer through the fisheye camera on the wearable device, and perform segmentation processing operation on the hand image to obtain a hand segmentation image. Then, the hand segmentation image can be mapped to a preset background to obtain a mapped hand segmentation image. Further, the mapped hand segmentation image can be subjected to fisheye distortion operation to obtain a hand segmentation image after fisheye distortion, and the corresponding target gesture frame can be determined based on the hand segmentation image after fisheye distortion. In this way, the pre-trained model can be trained using the target gesture frame to obtain a gesture recognition model. The present application can collect hand images through a fisheye camera with low power consumption and a large viewing angle range, and on the basis of improving the collection efficiency of hand images, the collection cost of hand images is reduced. Furthermore, after obtaining the hand image through the fisheye camera, the hand image is sequentially subjected to segmentation processing operation and background mapping operation. Since the hand segmentation image is mapped to a variety of preset backgrounds, the richness of the hand segmentation image is improved, thereby improving the generalization ability of the target gesture frame. Furthermore, the above method directly determines the corresponding target gesture frame based on the hand segmentation image after fisheye distortion, that is, the target gesture frame can be obtained by only one annotation. Therefore, on the basis of improving the generation efficiency of the target gesture frame, the stability of the target gesture frame is also improved, and the annotation cost of the gesture frame is reduced.

[0059] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of obtaining a hand segmentation image provided by an embodiment of the present application. Figure 2 As shown, the segmentation operation on the hand image can be implemented through steps S111 to S112 to obtain a hand segmentation image.

[0060] Step S111: performing distortion correction processing on the hand image to obtain a distortion-corrected hand image; wherein the hand image includes a background and a gesture image.

[0061] It is understandable that after the hand image is captured by the fisheye camera, the hand image will show a distortion in the central area and a stretching effect in the edge area due to the special design of the fisheye camera lens. This distortion causes the straight lines in the hand image to no longer be straight, but to present a curved shape. Therefore, in order to correct the hand image to a normal image and then segment it into a hand segmentation image, the hand image can be subjected to distortion correction processing to obtain a distortion-corrected hand image.

[0062] It should be noted that the present application does not limit the method of distortion correction processing, for example, including mathematical model method, calibration method, grid transformation method and feature point matching method to achieve distortion correction processing. The present application takes the calibration method as an example to illustrate the distortion correction processing method. Specifically, the calibration method can be achieved by placing a specific pattern or calibration plate on the fisheye camera, taking multiple images of hands with consistent distortion, thereby obtaining the parameters required for distortion correction, and performing distortion correction accordingly.

[0063] Optionally, in order to improve the degree of generalization, a black curtain can be used as the background, and a fisheye camera can be used to capture a hand image against the black curtain background; then, only a complex background image can be captured through the fisheye camera, and then the distortion correction processing method can be used to perform distortion correction on the hand image and the complex background image respectively, thereby obtaining a distortion-corrected hand image.

[0064] Optionally, the distortion-corrected hand image may be framed and cropped to narrow the range of gesture recognition, thereby improving gesture recognition efficiency.

[0065] In an embodiment of the present application, the hand image can be subjected to distortion correction processing to obtain a distortion-corrected hand image, wherein the distortion-corrected image includes a background and a gesture image. By correcting the distortion of the hand image to a normal image, it can be further segmented into a hand segmentation image containing only the gesture image.

[0066] Step S112: performing a cutout operation on the distortion-corrected hand image to obtain a hand segmentation image; wherein the hand segmentation image includes a gesture image.

[0067] Specifically, the background may be cut out from the distortion-corrected hand image to obtain a hand segmentation image including only the gesture image.

[0068] In the embodiment of the present application, the distortion-corrected hand image can be cut out to obtain a hand segmentation image including only the gesture image. In this way, a corresponding gesture recognition frame can be obtained based on the hand segmentation image to improve gesture recognition efficiency.

[0069] See also Figure 3 as well as Figure 4 , Figure 3 is another schematic diagram of a process for obtaining a hand segmentation image provided by an embodiment of the present application; Figure 4 FIG. 1 is another schematic diagram of a scene for obtaining a hand segmentation image provided by an embodiment of the present application. Figure 3 As shown, the hand image can be segmented through steps S113 to S118 to obtain a hand segmentation image. Figure 4 As shown, Figure 4The images from left to right are named image, mask1, mask2, mask3, and mask4.

[0070] Step S113: performing distortion correction processing on the hand image to obtain a distortion-corrected hand image.

[0071] Step S114: performing a threshold segmentation operation on the distortion-corrected hand image to obtain a first hand image.

[0072] The first hand image is a hand image segmented by threshold value.

[0073] like Figure 4 As shown in FIG. 1 , image is the hand image after distortion correction, and mask1 is the first hand image. The background pixel value of the first hand image is 0, and the pixel value of the gesture image is 255.

[0074] Specifically, after the distortion correction processing is performed on the hand image to obtain the distortion-corrected hand image, a threshold segmentation operation may be performed on the hand image to preliminarily segment the hand gesture image from the background based on pixel values.

[0075] It should be noted that the threshold segmentation operation is an image processing method for segmenting an image into different regions with different brightness or colors. A corresponding threshold is calculated for each region, and then each region is binarized according to the calculated threshold, and pixels are divided into a target region and a background region. Therefore, the embodiment of the present application can obtain a first hand image by performing a threshold segmentation operation on the distortion-corrected hand image.

[0076] Furthermore, in a grayscale image, the pixel value represents the brightness level of each pixel in the image. For example, a pixel value of 0 represents the darkest color (usually black), while a pixel value of 255 represents the brightest color (usually white). Therefore, ensuring that the background pixel value of the first hand image is 0 and the pixel value of the gesture image is 255 is conducive to distinguishing the background from the gesture image, and further facilitates image analysis and processing.

[0077] Step S115: Detect the first connected domain in the first hand image, and determine the image corresponding to the first connected domain with the largest area as the second hand image.

[0078] like Figure 4 As shown, mask2 is the second hand image.

[0079] Since there may be scattered noise in the first hand image, in order to improve the efficiency of subsequent image processing, the first connected domain in the first hand image may be detected, and the image corresponding to the first connected domain with the largest area is determined as the second hand image. In this way, the influence of scattered noise on the gesture image can be avoided.

[0080] Step S116: performing a black-and-white flipping operation on the second hand image, detecting a second connected domain of the second hand image after the black-and-white flipping operation, and determining a third hand image based on the second connected domain.

[0081] Among them, the background pixel value of the third hand image is 0, and the pixel value of the gesture image is 255.

[0082] Furthermore, the background in the third hand image is an image corresponding to the second connected domain with the largest area; and the gesture image in the third hand image is other images except the background.

[0083] Specifically, the second hand image may be subjected to a black-and-white flip operation to prevent the gesture image with a lower grayscale value in the second hand image from being detected as the background. It is understandable that the background pixel value of the second hand image is 0, and the pixel value of the gesture image is 255. Therefore, the background pixel value of the second hand image after the black-and-white flip operation is 255, and the pixel value of the gesture image is 0. Further, the second connected domain of the second hand image after the black-and-white flip operation may be detected, and the image corresponding to the second connected domain with the largest area may be determined as the background in the third hand image, and the images corresponding to other connected domains except the background may be determined as the gesture image in the third hand image, wherein the background pixel value of the third hand image is 0, and the pixel value of the gesture image is 255.

[0084] Step S117: performing an erosion operation on the third hand image to obtain a fourth hand image.

[0085] Step S118: performing a cutout operation on the fourth hand image to obtain a hand segmentation image.

[0086] like Figure 4 As shown, mask3 is the third hand image, and mask4 is the fourth hand image.

[0087] Specifically, the third hand image may be subjected to an erosion operation to obtain a fourth hand image after the erosion operation. The erosion operation can prevent the problem of black edges appearing around the hand gesture image obtained by cutting out.

[0088] Furthermore, a matting operation may be performed on the fourth hand image after the corrosion operation to obtain a more accurate hand segmentation image.

[0089] See also Figure 5 , Figure 5 is a schematic diagram of a scene of a hand segmentation image provided by an embodiment of the present application. Figure 5 The left image is the hand segmentation image obtained without corrosion operation; Figure 5 The image on the right is the hand segmentation image obtained after the corrosion operation. Figure 5As shown, after the third hand image is corroded, the obtained hand segmentation image has no black edges. However, the hand segmentation image obtained without the third hand image being corroded has black edges. Therefore, the present application can significantly prevent the problem of black edges appearing around the gesture image by corroding the third hand image.

[0090] In an embodiment of the present application, the hand image can be subjected to distortion correction processing, threshold segmentation operation, connected domain detection operation, black-and-white flipping operation, corrosion operation, and cutout operation in sequence to greatly improve the background richness, thereby ensuring the diversity of the image and obtaining a more stable gesture recognition frame.

[0091] See also Figure 6 , Figure 6 Schematic diagram of a process of mapping a hand segmentation image to a preset background provided by an embodiment of the present application. Figure 6 As shown, the hand segmentation image can be mapped to the preset background through steps S121 to S124.

[0092] Step S121: selecting a reference point in the hand segmentation image, and determining first coordinate information of the reference point in the hand segmentation image and second coordinate information of the reference point in the preset background.

[0093] The reference point may be a specific landmark position in the hand segmentation image, such as a finger joint or a palm center. Further, the number of the reference point may be one or more, which is not limited in the present application.

[0094] Specifically, for each reference point, first coordinate information and second coordinate information thereof in the hand segmentation image and in the preset background may be determined respectively.

[0095] Step S122: Subtract the first coordinate information from the second coordinate information to obtain a position offset of the reference point.

[0096] Step S123: Based on the position offset, determine the third coordinate information of each coordinate point in the hand segmentation image on the preset background.

[0097] Step S124: Mapping the hand segmentation image to a preset background based on a plurality of third coordinate information.

[0098] The first coordinate information is the coordinate information of the reference point in the hand segmentation image; the second coordinate information is the coordinate information of the reference point in the preset background.

[0099] Specifically, for each reference point, the first coordinate information and the second coordinate information corresponding to the reference point are subtracted. ，To obtain the position offset. The difference operation is to subtract the second coordinate information from the first coordinate information; or, the second coordinate information from the first coordinate information, which is not limited in the present application.

[0100] It can be understood that the position offset obtained after the difference can be used to measure the relative position difference between the reference point in the hand segmentation image and the preset background image. That is, based on the position offset calculated as above, each coordinate point in the hand segmentation image can be processed to determine its position in the preset background image. For example, an interpolation method can be used to calculate the coordinate position of each coordinate point in the hand image in the preset background image according to the position offset to obtain the third coordinate information. In this way, the hand segmentation image can be mapped to the preset background image based on a number of third coordinate information, that is, the hand segmentation image can be mapped to the preset background.

[0101] In an embodiment of the present application, the offset of the hand segmentation image mapped to the preset background can be obtained based on the reference point in the hand segmentation image, thereby mapping the hand segmentation image to the preset background for automatically generating a more stable gesture recognition frame.

[0102] Please continue reading Figure 7 as well as Figure 8 , Figure 7 It is a schematic diagram of a process of obtaining a hand segmentation image after fisheye distortion provided by an embodiment of the present application; Figure 8 Schematic diagram of an isometric model provided in an embodiment of the present application. Figure 7 As shown, the fisheye distortion operation can be performed on the mapped hand segmentation image through steps S131 to S132 to obtain the hand segmentation image after fisheye distortion.

[0103] Step S131: Obtain an isometric model.

[0104] Step S132: Based on the isometric model, a fisheye distortion operation is performed on the mapped hand segmentation image to obtain a hand segmentation image after fisheye distortion.

[0105] For ease of understanding, this application provides the following examples:

[0106] like Figure 8 As shown in the figure, AD is the actual length of the object before distortion, that is, the hand segmentation image after mapping; C is the midpoint of AD; E is the fisheye lens; FI is the distorted length of the image formed by the fisheye camera, that is, the hand segmentation image after fisheye distortion; G is the midpoint of FI; assuming that there is a coordinate point B on AD, then its coordinate point photographed by the fisheye camera is H. According to the isometric model, there is formula 1:

[0107] ∠BEC / ∠AEC=GH / GI.

[0108] In actual calculation, assuming that the image before fisheye distortion operation is img1, the image after fisheye distortion operation is img2, the height and width of img1 are h1 and w1 respectively, and the height and width of img2 are h2 and w2 respectively, then the coordinate value of the midpoint in img1 is The midpoint coordinates in img2 are

[0109] Assume CE=z, a coordinate point (x1, y1) in img1 corresponds to a coordinate point (x2, y2) in img2, then:

[0110] In the x-axis direction, AB=x1,HI=x2,

[0111]

[0112]

[0113] According to formula 1, we can get:

[0114]

[0115] Furthermore, we can obtain:

[0116]

[0117] Similarly, y2 can be obtained according to the above formula.

[0118] In this way, every coordinate point on img1 can find a corresponding point on img2.

[0119] That is, the corresponding coordinate point of each coordinate point of the mapped hand segmentation image in the hand segmentation image after fisheye distortion can be determined based on the isometric model, and the hand segmentation image after fisheye distortion can be determined based on several corresponding coordinate points.

[0120] Optionally, based on the above embodiment, the corresponding target gesture frame is determined based on the hand segmentation image after fisheye distortion, including: determining the maximum value, minimum value in the x-axis direction, and the maximum value and minimum value in the y-axis direction of each coordinate information corresponding to the hand segmentation image after fisheye distortion; taking the coordinate information corresponding to the maximum value and minimum value in the x-axis direction and the y-axis direction as vertices, respectively, to obtain a closed area, and determining the closed area as the target gesture frame.

[0121] Specifically, each coordinate information constituting the hand segmentation image after fisheye distortion can also be determined by the above method according to the isometric model, and then the maximum and minimum values ​​of each coordinate information in the x-axis direction and the y-axis direction can be determined. In this way, the coordinate information corresponding to the maximum and minimum values ​​in the x-axis direction and the y-axis direction can be used as vertices, and a closed area composed of vertices can be obtained. The closed area is determined as the target gesture frame, which is used to implement the training of the recognition model based on the target gesture frame.

[0122] It is understandable that in the above distortion simulation process based on the isometric model, there are only three variables: z, h2 and w2. Therefore, in the actual mapping process, these three parameters can be randomly selected within a certain range to simulate a variety of fisheye distortion processes, thereby generating a variety of distorted images. Therefore, it is possible to further obtain several target gesture recognition frames based on multiple distorted images for training the gesture recognition model.

[0123] In the embodiment of the present application, the mapped hand segmentation image can be subjected to fisheye distortion operation based on the isometric model to obtain the hand segmentation image after fisheye distortion. In addition, the target gesture frame can also be determined based on the isometric model to ensure the stability of the target gesture frame annotation and avoid artificial noise caused by multiple annotations of the same gesture.

[0124] See also Fig. 9 , Fig. 9 1 is a schematic block diagram of a hand image annotation device provided in an embodiment of the present application. The hand image annotation device can be configured in a server to execute the hand image annotation method described above.

[0125] like Fig. 9 As shown, the hand image annotation device 200 includes: a collection module 201, a mapping module 202, a determination module 203, and a training module 204.

[0126] The acquisition module 201 is used to acquire a hand image of the wearer through a fisheye camera, and perform a segmentation processing operation on the hand image to obtain a hand segmentation image, wherein the hand segmentation image includes a gesture image;

[0127] A mapping module 202 is used to map the hand segmentation image to a preset background to obtain a mapped hand segmentation image;

[0128] The determination module 203 is used to perform a fisheye distortion operation on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image, and determine a corresponding target gesture frame based on the fisheye distorted hand segmentation image.

[0129] The acquisition module 201 is also used to perform distortion correction processing on the hand image to obtain a distortion-corrected hand image; wherein the hand image includes a background and a gesture image; and perform a cutout operation on the distortion-corrected hand image to obtain a hand segmentation image; wherein the distortion-corrected hand segmentation image includes the gesture image.

[0130] The acquisition module 201 is also used to perform a threshold segmentation operation on the hand image after distortion correction to obtain a first hand image; wherein the background pixel value of the first hand image is 0, and the pixel value of the gesture image is 255; detect the first connected domain in the first hand image, and determine the image corresponding to the first connected domain with the largest area as the second hand image; perform a black-and-white flip operation on the second hand image, detect the second connected domain of the second hand image after the black-and-white flip operation, and determine the third hand image based on the second connected domain; wherein the background pixel value of the third hand image is 0, and the pixel value of the gesture image is 255; perform an erosion operation on the third hand image to obtain a fourth hand image; perform a cutout operation on the fourth hand image to obtain the hand segmentation image.

[0131] The mapping module 202 is also used to select a reference point in the hand segmentation image, and respectively determine the first coordinate information of the reference point in the hand segmentation image and the second coordinate information of the preset background; subtract the first coordinate information from the second coordinate information to obtain the position offset of the reference point; based on the position offset, determine the third coordinate information of each coordinate point constituting the hand segmentation image in the preset background; based on a number of the third coordinate information, map the hand segmentation image to the preset background.

[0132] The determination module 203 is further used to obtain an isometric model; based on the isometric model, the mapped hand segmentation image is subjected to a fisheye distortion operation to obtain the hand segmentation image after the fisheye distortion.

[0133] The determination module 203 is also used to determine the maximum value, minimum value in the x-axis direction and the maximum value, minimum value in the y-axis direction of each coordinate information corresponding to the hand segmentation image after fisheye distortion; the coordinate information corresponding to the maximum value and minimum value in the x-axis direction and the y-axis direction are respectively used as vertices to obtain a closed area, and the closed area is determined as the target gesture frame.

[0134] The training module 204 is further used to perform a feature extraction operation on the target gesture frame to obtain a corresponding feature vector; and analyze the feature vector to obtain the gesture recognition result.

[0135] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0136] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0137] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Fig.10 The wearable device shown is running on the

[0138] See also Fig.10 , Fig.10 is a schematic diagram of a wearable device provided in an embodiment of the present application. The wearable device may be a server.

[0139] like Fig.10 As shown, the wearable device 400 includes a processor 401, a memory 402 and a network interface connected via a system bus, wherein the memory 402 may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0140] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor 401 can execute any one of the hand image annotation methods.

[0141] The processor 401 is used to provide computing and control capabilities to support the operation of the entire wearable device 400.

[0142] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the hand image labeling methods.

[0143] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that the structure of the wearable device is only a block diagram of a portion of the structure related to the present application scheme, and does not constitute a limitation on the wearable device 400 to which the present application scheme is applied. The specific wearable device 400 may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] It should be understood that the processor 401 may be a central processing unit (CPU), and the processor 401 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0145] Among them, in some embodiments, the processor 401 is used to run a computer program stored in the memory to implement the following steps: collecting the wearer's hand image through the fisheye camera, performing a segmentation processing operation on the hand image to obtain a hand segmentation image; mapping the hand segmentation image to a preset background to obtain a mapped hand segmentation image; performing a fisheye distortion operation on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image, and determining the corresponding target gesture frame based on the fisheye distorted hand segmentation image.

[0146] In some embodiments, the processor 401 is further used to perform distortion correction processing on the hand image to obtain a distortion-corrected hand image; wherein the hand image includes a background and a gesture image; and perform a cutout operation on the distortion-corrected hand image to obtain a hand segmentation image; wherein the distortion-corrected hand segmentation image includes the gesture image.

[0147] In some embodiments, the processor 401 is also used to perform a threshold segmentation operation on the distortion-corrected hand image to obtain a first hand image; wherein the background pixel value of the first hand image is 0, and the pixel value of the gesture image is 255; detect the first connected domain in the first hand image, and determine the image corresponding to the first connected domain with the largest area as the second hand image; perform a black-and-white flip operation on the second hand image, detect the second connected domain of the second hand image after the black-and-white flip operation, and determine a third hand image based on the second connected domain; wherein the background pixel value of the third hand image is 0, and the pixel value of the gesture image is 255; perform an erosion operation on the third hand image to obtain a fourth hand image; perform a cutout operation on the fourth hand image to obtain the hand segmentation image.

[0148] In some embodiments, the processor 401 is also used to select a reference point in the hand segmentation image, and respectively determine the first coordinate information of the reference point in the hand segmentation image and the second coordinate information of the preset background; subtract the first coordinate information from the second coordinate information to obtain the position offset of the reference point; based on the position offset, determine the third coordinate information of each coordinate point constituting the hand segmentation image in the preset background; based on a number of the third coordinate information, map the hand segmentation image to the preset background.

[0149] In some implementations, the processor 401 is further configured to obtain an isometric model; based on the isometric model, perform a fisheye distortion operation on the mapped hand segmentation image to obtain the fisheye distorted hand segmentation image.

[0150] In some embodiments, the processor 401 is also used to determine the maximum value, minimum value in the x-axis direction and the maximum value, minimum value in the y-axis direction of each coordinate information corresponding to the hand segmentation image after fisheye distortion; the coordinate information corresponding to the maximum value and minimum value in the x-axis direction and the y-axis direction are respectively used as vertices to obtain a closed area, and the closed area is determined as the target gesture frame.

[0151] In some implementations, the processor 401 is further configured to perform a feature extraction operation on the target gesture frame to obtain a corresponding feature vector; and analyze the feature vector to obtain the gesture recognition result.

[0152] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any one of the hand image annotation methods provided in the embodiments of the present application is implemented.

[0153] The computer-readable storage medium may be an internal storage unit of the wearable device described in the foregoing embodiment, such as a hard disk or memory of the wearable device. The computer-readable storage medium may also be an external storage device of the wearable device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the wearable device.

[0154] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A hand image annotation method, characterized in that: The method comprises: The fisheye camera on the wearable device collects a hand image of the wearer, and performs a segmentation processing operation on the hand image to obtain a hand segmentation image; Mapping the hand segmentation image to a preset background to obtain a mapped hand segmentation image; A fisheye distortion operation is performed on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image, and a corresponding target gesture frame is determined based on the fisheye distorted hand segmentation image.

2. The method according to claim 1, characterized in that The segmenting operation is performed on the hand image to obtain a hand segmentation image, including: Performing distortion correction processing on the hand image to obtain a distortion-corrected hand image; wherein the distortion-corrected hand image includes a background and a gesture image; A cutout operation is performed on the distortion-corrected hand image to obtain a hand segmentation image; wherein the hand segmentation image includes the gesture image.

3. The method according to claim 2, characterized in that Before performing a cutout operation on the distortion-corrected hand image to obtain a hand segmentation image, the method further includes: Performing a threshold segmentation operation on the distortion-corrected hand image to obtain a first hand image; wherein the background pixel value of the first hand image is 0, and the pixel value of the gesture image is 255; Detecting a first connected domain in the first hand image, and determining an image corresponding to the first connected domain having the largest area as the second hand image; Performing a black-white flip operation on the second hand image, detecting a second connected domain of the second hand image after the black-white flip operation, and determining a third hand image based on the second connected domain; wherein the background pixel value of the third hand image is 0, and the pixel value of the gesture image is 255; Performing an erosion operation on the third hand image to obtain a fourth hand image; The step of performing a cutout operation on the distortion-corrected hand image to obtain a hand segmentation image includes: A cutout operation is performed on the fourth hand image to obtain the hand segmentation image.

4. The method according to claim 1, characterized in that: Mapping the hand segmentation image to a preset background includes: Selecting a reference point in the hand segmentation image, and respectively determining first coordinate information of the reference point in the hand segmentation image and second coordinate information of the reference point in the preset background; Subtract the first coordinate information from the second coordinate information to obtain a position offset of the reference point; Based on the position offset, determine the third coordinate information of each coordinate point in the hand segmentation image on the preset background; Based on the third coordinate information, the hand segmentation image is mapped to a preset background.

5. The method according to claim 1, characterized in that The step of performing a fisheye distortion operation on the mapped hand segmentation image to obtain a fisheye distorted hand segmentation image includes: Get an isometric model; Based on the isometric model, a fisheye distortion operation is performed on the mapped hand segmentation image to obtain the fisheye distorted hand segmentation image.

6. The method according to claim 5, characterized in that The determining a corresponding target gesture frame based on the hand segmentation image after fisheye distortion includes: Determine the maximum value and the minimum value of each coordinate information corresponding to the hand segmentation image after the fisheye distortion in the x-axis direction and the maximum value and the minimum value in the y-axis direction; The coordinate information corresponding to the maximum value and the minimum value in the x-axis direction and the y-axis direction are respectively used as vertices to obtain a closed area, and the closed area is determined as the target gesture frame.

7. The method according to claim 1, characterized in that After determining the corresponding target gesture frame based on the hand segmentation image after fisheye distortion, the method further includes: Performing a feature extraction operation on the target gesture frame to obtain a corresponding feature vector; The feature vector is analyzed to obtain a gesture recognition result.

8. A hand image annotation device, characterized in that: include: An acquisition module, wherein the acquisition module is used to acquire a hand image of the wearer through a fisheye camera on the wearable device, and perform a segmentation processing operation on the hand image to obtain a hand segmentation image; A mapping module, wherein the mapping module is used to map the hand segmentation image to a preset background to obtain a mapped hand segmentation image; A determination module is used to perform a fisheye distortion operation on the mapped hand segmentation image to obtain a hand segmentation image after fisheye distortion, and determine a corresponding target gesture frame based on the hand segmentation image after fisheye distortion.

9. A wearable device, characterized in that: include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the hand image annotation method according to any one of claims 1 to 7 by running the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the hand image annotation method according to any one of claims 1 to 7.

Images