Hand data set generation method and electronic equipment
The semantic segmentation model is used to perform hand area segmentation and boundary noise smoothing processing on the real image, which solves the problem of limited background types and white edges and fragments in the hand data set generation method in the prior art, and improves the quality of the data set and the detection performance of the AI model.
Patent Information
- Application Number
- CN202311131826.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-01
- Publication Date
- 2025-03-11
AI Technical Summary
In the existing hand dataset generation methods, the background types of real images captured by the motion capture system are limited, and there is a gap between the data synthesized by 3D modeling and real images, which affects the accuracy of the AI model extracting key points at the hand. At the same time, there are white edges and fragments in the hand area divided by artificial intelligence, resulting in poor quality of the data set.
The semantic segmentation model is used to segment the real image in the hand area, filter the mask image of the target hand area, identify the hand boundary line, and smooth the mask image of the initial hand synthesized image and the boundary noise mask image to generate a high-quality hand data set.
The quality of the generated hand data set is improved, and the problem of white edges and microblocks in the segmented hand area is solved, which enhances the authenticity and diversity of the data set, and thus improves the detection performance of the AI model for extracting hand key points.
Smart Images
Figure CN119600640A_ABST
Abstract
Description
Background Art
[0002] The existing hand training data generation scheme mainly includes two methods: one is to shoot the hand with the camera of the motion capture system to obtain the real image of the hand, and record the 3D position coordinates and 2D projection position coordinates of each key point of the hand during the shooting process to obtain a hand data set marked with position coordinates. The other is to use the 3D position coordinates of the existing hand key points and 3D modeling technology to create a 3D model of the hand in a virtual scene, and then generate an image with marked 3D position coordinates by shooting the image of the 3D model of the hand in the virtual scene to obtain a hand data set.
[0003] However, the hand data set obtained based on the motion capture system can obtain real images of the hands taken by the camera. However, due to the complexity of setting up the motion capture system, it is generally not convenient to move, so the types of backgrounds that can be captured are quite limited. In the process of training the AI model for hand key point extraction, more different types of background data are often crucial to improving the detection performance of the AI model for hand key point extraction. Moreover, although the data synthesized based on 3D modeling can obtain very diverse scenes and accurate annotation values, there is still a certain gap between the background images and human hand images generated by the modeling and the real images taken by the real camera, which has a certain impact on the accuracy of the AI model for hand key point extraction.
[0004] In this case, a new method is proposed in the prior art, which is to use the real image captured by the motion capture system to obtain the area that needs to be detected (for example, the outline of a human hand). The area that needs to be detected is then cropped out from the original image, and then pasted onto background images of different scenes. In this way, both the authenticity of the image taken by the camera and the diversity of the background can be taken into account. However, there is also the problem of how to crop out the hand area. Based on the existing open source artificial intelligence general segmentation model, the outline of the human hand can be roughly segmented, but the accuracy of artificial intelligence segmentation is still lower than that of manual segmentation by humans, and there are often problems such as tiny fragments and white edges. For example, Figure 1 As shown, from Figure 1 As can be seen from the figure, there are white edges and fragments on the contour edge of the hand in the synthesized hand image, which cannot be well integrated with the background image. Moreover, these edges cannot be removed by simple filtering, which will result in poor quality of the generated hand dataset. Summary of the invention
[0005] The present application provides a method for generating a hand data set and an electronic device, which are used to improve the quality of the generated hand data set.
[0006] In a first aspect, an embodiment of the present application provides a method for generating a hand data set, the method comprising:
[0007] Using a semantic segmentation model to segment a real image containing a hand into a hand region, and obtaining at least one mask image of the hand region;
[0008] Based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand area, the mask image of each hand area is screened to obtain the mask image of the target hand area;
[0009] Using the mask image of the target hand region, identifying a hand boundary line of the target hand region, to obtain a mask image of the hand boundary line;
[0010] For any background image in the initial hand data set, an initial hand composite image is obtained according to the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region; and
[0011] Obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image; and
[0012] Smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image;
[0013] Based on the target hand composite image corresponding to each background image in the initial hand data set, a target hand data set is obtained.
[0014] A second aspect of the present application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;
[0015] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:
[0016] Using a semantic segmentation model to segment a real image containing a hand into a hand region, and obtaining at least one mask image of the hand region;
[0017] Based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand area, the mask image of each hand area is screened to obtain the mask image of the target hand area;
[0018] Using the mask image of the target hand region, identifying a hand boundary line of the target hand region, to obtain a mask image of the hand boundary line;
[0019] For any background image in the initial hand data set, an initial hand composite image is obtained according to the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region; and
[0020] Obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image; and
[0021] Smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image;
[0022] Based on the target hand composite image corresponding to each background image in the initial hand data set, a target hand data set is obtained.
[0023] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.
[0024] In the above-mentioned embodiment of the present application, the mask image of the target hand area is obtained by screening the mask images of the hand area obtained after the hand area segmentation of the real image, and then the mask image of the target hand area is used to identify the hand boundary line of the target hand area to obtain the mask image of the hand boundary line, and the corresponding initial hand composite image is obtained by using the mask image of the target hand area to segment the real image to obtain the hand area image and any background image in the initial hand data set, and then the mask image of the boundary noise in the initial hand composite image is obtained by the initial hand composite image and the mask image of the hand boundary line, and the boundary noise in the initial hand composite image is smoothed according to the mask image of the boundary noise to obtain the target hand composite image, and finally the target hand data set is obtained based on the target hand composite image corresponding to each background image in the initial hand data set. Therefore, in the embodiment of the present application, the boundary line of the hand area in the image can be obtained, and then the boundary noise in the image can be determined by the boundary line, and the boundary noise is smoothed to obtain the target hand composite image, and the target hand data set is obtained based on the target hand composite image. Therefore, the problem of white edges and micro-blocks in the segmented hand area in the prior art is solved, and the quality of the generated hand data set is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0026] Figure 1 A schematic diagram of a composite image in the prior art provided by an embodiment of the present application is exemplarily shown;
[0027] Figure 2 One of the flowcharts of the method for generating a hand data set provided in an embodiment of the present application is exemplarily shown;
[0028] Figure 3 The flowchart of the mask image for determining the hand boundary line provided by the embodiment of the present application is exemplarily shown;
[0029] Figure 4 A schematic diagram of a mask image of a hand boundary line provided in an embodiment of the present application is exemplarily shown;
[0030] Figure 5 The following is a schematic diagram of a process for determining an initial hand composite image provided by an embodiment of the present application;
[0031] Figure 6 The schematic diagram of brightness adjustment provided by the embodiment of the present application is exemplarily shown;
[0032] Figure 7 The flowchart of determining the mask image of the boundary noise point in the initial hand composite image provided by the embodiment of the present application is exemplarily shown;
[0033] Figure 8 A schematic diagram of a mask image for boundary noise provided in an embodiment of the present application is exemplarily shown;
[0034] Fig. 9 A schematic diagram of a target hand composite image provided by an embodiment of the present application is exemplarily shown;
[0035] Fig.10 The second flowchart of the method for generating a hand data set provided in an embodiment of the present application is exemplarily shown;
[0036] Fig.11 A schematic diagram of a device for generating a hand data set provided in an embodiment of the present application is exemplarily shown;
[0037] Fig.12 The hardware structure diagram of the electronic device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION
[0038] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0039] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.
[0040] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0041] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.
[0042] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.
[0043] The idea of the embodiments of the present application is summarized below.
[0044] At present, the method for generating hand datasets is to use the real image captured by the motion capture system to obtain the area to be detected. Then the area to be detected is cropped out from the original image, and then it is pasted onto the background image of different scenes. This can take into account both the authenticity of the image captured by the camera and the diversity of the background. However, there is also a problem with how to crop out the hand area. Based on the existing open source artificial intelligence general segmentation model, the outline of the human hand can be roughly segmented, but the accuracy of artificial intelligence segmentation is still far behind the effect of manual segmentation by humans, and there are often problems such as tiny fragments and white edges. These edges cannot be removed by simple filtering, which will result in poor quality of the generated hand dataset.
[0045] Based on the problem in the prior art that the segmented hand area may have tiny fragments and white edges, resulting in poor quality of the generated hand data set. The embodiment of the present application provides a method for generating a hand data set, by screening the mask images of the hand area obtained after segmenting the hand area of the real image to obtain the mask image of the target hand area, and then using the mask image of the target hand area to identify the hand boundary line of the target hand area to obtain the mask image of the hand boundary line, and by using the mask image of the target hand area to segment the real image to obtain the hand area image and any background image in the initial hand data set to obtain the corresponding initial hand composite image, and then by using the initial hand composite image and the mask image of the hand boundary line to obtain the mask image of the boundary noise in the initial hand composite image, and according to the mask image of the boundary noise, the boundary noise in the initial hand composite image is smoothed to obtain the target hand composite image, and finally based on the target hand composite image corresponding to each background image in the initial hand data set, the target hand data set is obtained. Therefore, in the embodiment of the present application, the boundary line of the hand area in the image can be obtained, and then the boundary noise in the image can be determined by using the boundary line, and the boundary noise is smoothed to obtain the target hand composite image, and the target hand data set is obtained based on the target hand composite image. Therefore, the problem of white edges and micro-blocks in the segmented hand area in the prior art is solved, and the quality of the generated hand data set is improved. Below, the method for generating the hand data set in the embodiment of the present application is described in detail in conjunction with the accompanying drawings.
[0046] like Figure 2 FIG. 1 is a flow chart of a method for generating a hand data set, which may include the following steps:
[0047] Step 201: segmenting a real image containing a hand into a hand region using a semantic segmentation model to obtain at least one mask image of the hand region;
[0048] The real image in the embodiment of the present application is obtained by the motion capture system, and the 3D position coordinates and 2D projection position coordinates of each hand key point in the real image are obtained by the motion capture system in the embodiment of the present application. And the semantic segmentation model used in the embodiment of the present application is the SAM model (segment anything model). One or more specified hand key points are input into the SAM model, and then the SAM model obtains one or more mask images of the hand area according to the input hand key points.
[0049] It should be noted that the semantic segmentation model described above is only used for illustration and does not limit the semantic segmentation model in the embodiments of the present application. The semantic segmentation model in the embodiments of the present application can be set according to actual conditions.
[0050] Step 202: based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand region, the mask image of each hand region is screened to obtain the mask image of the target hand region;
[0051] In one embodiment, step 202 may be specifically implemented as follows: for any one of the hand regions, based on the 2D projection position coordinates of the multiple designated hand key points among the hand key points and the mask image of the hand region, determine whether the positional relationship between the multiple designated hand key points and the hand region all meets the designated conditions; if so, determine that the hand region is the target hand region, and determine the mask image of the hand region as the mask image of the target hand region. If not, delete the hand region.
[0052] The specified conditions include that the hand key point is inside the hand area, or that the hand key point is not inside the hand area and the edge distance between the hand key point and the hand area is less than a specified distance.
[0053] In the embodiment of the present application, whether the positional relationships between the multiple specified hand key points and the hand area all meet the specified conditions is determined in the following manner:
[0054] For any hand area, the specified hand key points are traversed. For any traversed hand key point, the hand key point and the mask image of the hand area are input into the pointPolygonTest function of opencv to obtain whether the hand key point is inside the hand area and when the hand key point is not inside the hand area, the edge distance between the hand key point and the hand area is obtained.
[0055] Among them, the edge distance in the embodiment of the present application is the distance between the vertical line between the hand key point and the hand area.
[0056] Step 203: using the mask image of the target hand region to identify the hand boundary line of the target hand region, to obtain the mask image of the hand boundary line;
[0057] like Figure 3 As shown, it is a schematic diagram of a process of determining a mask image of a hand boundary line, which may include the following steps:
[0058] Step 301: performing morphological corrosion processing on the mask image of the target hand region to obtain the mask image of the target hand region after corrosion;
[0059] In the embodiment of the present application, the mask image of the target region is subjected to two morphological erosion processes, and the erosion process in the embodiment of the present application can be implemented by the erode function of opencv.
[0060] Step 302: obtaining a mask image of the corroded hand boundary line in the target hand region based on the corroded mask image;
[0061] In one embodiment, step 302 can be specifically implemented as follows: for any pixel point in the mask image of the target hand area, subtract the pixel value of the pixel point in the mask image of the target hand area from the pixel value of the pixel point in the mask image after the corrosion, so as to obtain the target pixel value of the pixel point in the mask image of the corroded hand boundary line; and obtain the mask image of the corroded hand boundary line according to the target pixel value of each pixel point.
[0062] In the embodiment of the present application, the pixel points with the same position in the mask image of the target hand area and the mask image after corrosion are confirmed as the same pixel point. Since the mask image of the target hand area is a closed polygonal structure, the corrosion operation will corrode from its outermost periphery to the inside, and the size of the mask image after corrosion is smaller than the size of the mask image of the target hand area. Therefore, at this time, the mask image of the target hand area minus the mask image after corrosion, the mask image of the corroded hand boundary line is obtained, that is, the binary image of the outermost circle of pixels in the mask image of the target hand area.
[0063] Step 303: performing edge smoothing on the mask image of the target hand region, and performing edge detection on the smoothed mask image to obtain a smoothed mask image of the hand boundary line;
[0064] In the embodiment of the present application, the edge smoothing method of the mask image of the target hand area is to perform Gaussian filtering on the mask image of the target hand area. And the edge detection method in the embodiment of the present application is performed using the Canny edge detection function of opencv. However, the edge smoothing method and the edge detection method in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the edge smoothing method and the edge detection method.
[0065] Step 304: performing morphological dilation processing on the smoothed mask image of the hand boundary line to obtain an intermediate mask image of the hand boundary line;
[0066] The morphological dilation processing in the embodiment of the present application is performed by using the erode function of opencv. However, the embodiment of the present application does not limit the dilation processing method, and the dilation processing method in the embodiment of the present application can be set according to actual conditions.
[0067] Step 305: Fusing the mask image of the eroded hand boundary line with the intermediate mask image of the hand boundary line to obtain the mask image of the hand boundary line.
[0068] In one embodiment, step 305 can be specifically implemented as follows: for any pixel point in the mask image of the corroded hand boundary line, the pixel value of the pixel point in the mask image of the corroded hand boundary line is added to the pixel value of the pixel point in the intermediate mask image of the hand boundary line to obtain the target pixel value of the pixel point, and based on the target pixel value of each pixel point, the mask image of the hand boundary line is obtained.
[0069] The mask image of the hand boundary line in the embodiment of the present application includes all unnatural image stitching boundary areas, which facilitates subsequent processing of the area. Figure 4 , which is a schematic diagram of a mask image of a hand boundary line. It can be seen from the image that the image contains unnatural boundaries at the outermost edge of the hand and unnatural boundaries inside the hand area.
[0070] Step 204: for any background image in the initial hand data set, an initial hand composite image is obtained according to the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region;
[0071] like Figure 5 As shown, a flow chart for determining an initial hand composite image may include the following steps:
[0072] Step 501: Obtaining a target brightness of the hand region image according to the grayscale value of each pixel in the hand region image and the grayscale value of each pixel in the background image;
[0073] In one embodiment, step 501 can be specifically implemented as follows: determining the median of the grayscale values of each pixel in the hand area image as the first intermediate brightness value; and determining the median of the grayscale values of each pixel in the background image as the second intermediate brightness value; and obtaining the target brightness based on the first intermediate brightness value and the second intermediate brightness value.
[0074] In the embodiment of the present application, the target brightness is obtained by the following method: the second intermediate brightness value is divided by the first intermediate brightness value to obtain the target brightness. The target brightness can be obtained by formula (1):
[0075]
[0076] Among them, P is the target brightness, p2 is the second intermediate brightness value, and p1 is the first intermediate brightness value.
[0077] Step 502: adjusting the brightness of the hand region image using the target brightness to obtain an adjusted hand region image;
[0078] In the embodiment of the present application, the target brightness is input into the convertScaleAbs function of opencv to adjust the brightness of the hand area image.
[0079] Step 503: Use the adjusted hand area image as a foreground image and perform image fusion with the background image to obtain the initial hand composite image.
[0080] like Figure 6 As shown, Figure 6 The left image is the hand composite image without brightness adjustment, and the right image is the initial hand composite image after brightness adjustment. Figure 6 It can be seen that the initial hand synthetic salient image after brightness adjustment is more consistent with the actual situation of the background image, and is also more consistent with the brightness that the camera may capture in the current background environment.
[0081] It should be noted that: in the embodiment of the present application, the adjusted hand area image is pasted into the background image to obtain the initial hand composite image. The image fusion method can be set according to the actual situation, and the embodiment of the present application does not limit the image fusion method.
[0082] Step 205: Obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image;
[0083] like Figure 7 As shown, a flow chart of determining a mask image of boundary noise points in an initial hand composite image may include the following steps:
[0084] Step 701: interpolating the initial hand composite image using the mask image of the hand boundary line to obtain an intermediate hand composite image;
[0085] In the embodiment of the present application, the mask image of the hand boundary line is used as the interpolation mask image, and the corresponding area in the initial hand composite image is linearly interpolated to obtain the intermediate hand composite image. The interpolation processing method in the embodiment of the present application can be implemented by the inpaint function of opencv. That is, the mask image of the hand boundary line and the initial hand composite image are input into the inpaint function to obtain the intermediate hand composite image.
[0086] It should be noted that the interpolation processing method described above is only used as an example, but does not limit the specific method of interpolation processing. The interpolation processing method in the embodiment of the present application can be set according to actual conditions.
[0087] Step 702: for any pixel point in the initial hand composite image, based on the pixel value of the pixel point in the initial hand composite image and the pixel value of the pixel point in the intermediate hand composite image, obtain the pixel difference of the pixel point;
[0088] In one embodiment, step 702 may be specifically implemented as follows: determining the absolute value of the difference between the pixel value of the pixel point in the initial hand composite image and the pixel value of the pixel point in the intermediate composite image as the difference value of the pixel point.
[0089] Step 703: if the pixel difference of the pixel point is greater than a specified threshold, the pixel point is determined as a boundary noise point;
[0090] It should be noted that the specified threshold in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specified threshold.
[0091] Step 704: Mark the boundary noise points in the preset binary image according to the position coordinates of the boundary noise points to obtain a mask image of the boundary noise points.
[0092] The pixel value of each pixel in the binary image preset in the embodiment of the present application is 0. That is, the binary image preset in the embodiment of the present application is a completely black image.
[0093] In one embodiment, step 704 may be specifically implemented as follows: for any boundary noise point, the pixel value of the pixel point in the binary image having the same position as the boundary noise point is set to a specified pixel value to obtain a mask image of the boundary noise point.
[0094] For example, Figure 8 , which is a schematic diagram of a mask image of boundary noise, and includes the locations of all unnatural noise points on the boundary of the initial hand synthetic image.
[0095] It should be noted that the designated pixel value in the embodiment of the present application is 255. However, the specific value of the designated pixel value can be set according to actual conditions, and the embodiment of the present application does not limit the designated pixel value.
[0096] Step 206: smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image;
[0097] In one embodiment, step 206 may be specifically implemented as follows: interpolating the initial hand composite image using the mask image of the boundary noise point to obtain the target hand composite image.
[0098] The interpolation processing method in the embodiment of the present application can be implemented by the inpaint function of opencv. That is, the mask image of the boundary noise and the initial hand synthesis image are input into the inpaint function to obtain the target hand synthesis image. For example, Fig. 9 As shown in the figure, the final target hand composite image is obtained. Figure 1 Compared with the synthetic image in the prior art, the synthetic image of the target hand in the embodiment of the present application has no white edges and fragments, and is integrated with the background image very naturally.
[0099] Step 207: Obtain a target hand data set based on the target hand composite image corresponding to each background image in the initial hand data set.
[0100] In one embodiment, step 207 may be specifically implemented as follows: each background image is replaced with a target hand synthetic image corresponding to each background image to obtain the target hand data set.
[0101] In order to further connect the methods in this application, such as Fig.10 FIG. 1 is a flow chart of a method for generating a hand data set in the present application, which may include the following steps:
[0102] Step 1001: Segment the hand region of a real image containing a hand using a semantic segmentation model to obtain at least one mask image of the hand region;
[0103] Step 1002: based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand region, the mask image of each hand region is screened to obtain the mask image of the target hand region;
[0104] Step 1003: performing morphological corrosion processing on the mask image of the target hand region to obtain the mask image of the target hand region after corrosion;
[0105] Step 1004: obtaining a mask image of the corroded hand boundary line in the target hand region based on the corroded mask image;
[0106] Step 1005: performing edge smoothing on the mask image of the target hand region, and performing edge detection on the smoothed mask image to obtain a smoothed mask image of the hand boundary line;
[0107] Step 1006: performing morphological dilation processing on the smoothed mask image of the hand boundary line to obtain an intermediate mask image of the hand boundary line;
[0108] Step 1007: fusing the mask image of the eroded hand boundary line with the intermediate mask image of the hand boundary line to obtain the mask image of the hand boundary line;
[0109] Step 1008: for any background image in the initial hand data set, the target brightness of the hand region image is obtained according to the grayscale value of each pixel in the hand region image and the grayscale value of each pixel in the background image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region;
[0110] Step 1009: adjusting the brightness of the hand region image using the target brightness to obtain an adjusted hand region image;
[0111] Step 1010: using the adjusted hand region image as a foreground image and fusing it with the background image to obtain the initial hand composite image;
[0112] Step 1011: performing interpolation processing on the initial hand composite image using the mask image of the hand boundary line to obtain an intermediate hand composite image;
[0113] Step 1012: for any pixel point in the initial hand composite image, based on the pixel value of the pixel point in the initial hand composite image and the pixel value of the pixel point in the intermediate hand composite image, obtain the pixel difference of the pixel point;
[0114] Step 1013: Determine the pixel points whose pixel difference is greater than a specified threshold as boundary noise points;
[0115] Step 1014: marking the boundary noise points in a preset binary image according to the position coordinates of the boundary noise points, and obtaining a mask image of the boundary noise points;
[0116] Step 1015: smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image;
[0117] Step 1016: Obtain a target hand data set based on the target hand composite image corresponding to each background image in the initial hand data set.
[0118] Based on the same inventive concept, the method for generating a hand data set disclosed above can also be implemented by a device for generating a hand data set. The effect of the device for generating a hand data set is similar to that of the aforementioned method, and will not be described in detail here.
[0119] Fig.11 The figure is a schematic diagram of the structure of a device for generating a hand data set according to an embodiment of the present disclosure.
[0120] like Fig.11 The hand data set generation device 1100 of the present disclosure may include a hand region segmentation module 1110, a hand region screening module 1120, a hand boundary line recognition module 1130, an image synthesis module 1140, a boundary noise point positioning module 1150, an image smoothing module 1160 and a target hand data set determination module 1170.
[0121] A hand region segmentation module 1110 is used to segment the hand region of a real image containing a hand using a semantic segmentation model to obtain a mask image of at least one hand region;
[0122] A hand region screening module 1120, configured to screen the mask images of the hand regions based on the 2D projection position coordinates of the hand key points in the real image and the mask images of the hand regions, to obtain the mask image of the target hand region;
[0123] A hand boundary line recognition module 1130 is used to recognize the hand boundary line of the target hand area by using the mask image of the target hand area to obtain the mask image of the hand boundary line;
[0124] An image synthesis module 1140 is used to obtain an initial hand synthesis image according to any background image in the initial hand data set and the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region;
[0125] A boundary noise point positioning module 1150 is used to obtain a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image;
[0126] An image smoothing module 1160 is used to smooth the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image;
[0127] The target hand data set determination module 1170 is used to obtain a target hand data set based on the target hand composite image corresponding to each background image in the initial hand data set.
[0128] In one embodiment, the hand region screening module 1120 is specifically used to:
[0129] For any one of the hand regions, based on the 2D projection position coordinates of multiple designated hand key points among the hand key points and the mask image of the hand region, determine whether the positional relationships between the multiple designated hand key points and the hand region all meet designated conditions, wherein the designated conditions include that the hand key point is inside the hand region, or that the hand key point is not inside the hand region and the edge distance between the hand key point and the hand region is less than a designated distance;
[0130] If so, the hand region is determined to be the target hand region, and the mask image of the hand region is determined to be the mask image of the target hand region.
[0131] In one embodiment, the hand boundary line recognition module 1130 is further used to:
[0132] Performing morphological corrosion processing on the mask image of the target hand region to obtain the mask image of the target hand region after corrosion; and obtaining a mask image of the corroded hand boundary line in the target hand region based on the mask image after corrosion; and,
[0133] Performing edge smoothing processing on the mask image of the target hand area, and performing edge detection on the smoothed mask image to obtain a smoothed mask image of the hand boundary line, and performing morphological dilation processing on the smoothed mask image of the hand boundary line to obtain an intermediate mask image of the hand boundary line;
[0134] The mask image of the eroded hand boundary line and the intermediate mask image of the hand boundary line are fused to obtain the mask image of the hand boundary line.
[0135] In one embodiment, the hand boundary line recognition module 1130 is specifically used to:
[0136] For any pixel point in the mask image of the target hand area, subtract the pixel value of the pixel point in the mask image of the target hand area from the pixel value of the pixel point in the mask image after the corrosion, so as to obtain the target pixel value of the pixel point in the mask image of the corroded hand boundary line;
[0137] According to the target pixel value of each pixel point, a mask image of the corroded hand boundary line is obtained.
[0138] In one embodiment, the image synthesis module 1140 is specifically used to:
[0139] Obtaining a target brightness of the hand area image according to a grayscale value of each pixel in the hand area image and a grayscale value of each pixel in the background image;
[0140] adjusting the brightness of the hand area image using the target brightness to obtain an adjusted hand area image;
[0141] The adjusted hand area image is used as a foreground image and fused with the background image to obtain the initial hand composite image.
[0142] In one embodiment, the image synthesis module 1140 is specifically used to:
[0143] Determine the median of the grayscale value of each pixel in the hand area image as the first intermediate brightness value; and
[0144] Determine the median of the grayscale value of each pixel in the background image as the second intermediate brightness value;
[0145] The target brightness is obtained according to the first middle brightness value and the second middle brightness value.
[0146] In one embodiment, the image synthesis module 1140 is further used to:
[0147] The second intermediate brightness value is divided by the first intermediate brightness value to obtain the target brightness.
[0148] In one embodiment, the boundary noise point positioning module 1150 is specifically used to:
[0149] Using the mask image of the hand boundary line to interpolate the initial hand composite image to obtain an intermediate hand composite image;
[0150] For any pixel point in the initial hand synthesis image, based on the pixel value of the pixel point in the initial hand synthesis image and the pixel value of the pixel point in the intermediate hand synthesis image, a pixel difference of the pixel point is obtained; and,
[0151] If the pixel difference of the pixel point is greater than a specified threshold, the pixel point is determined as a boundary noise point;
[0152] According to the position coordinates of each boundary noise point, the boundary noise point is marked in a preset binary image to obtain a mask image of the boundary noise point.
[0153] In one embodiment, the image smoothing module 1160 is specifically configured to:
[0154] The initial hand composite image is interpolated using the mask image of the boundary noise point to obtain the target hand composite image.
[0155] After introducing a method and device for generating a hand data set according to an exemplary embodiment of the present invention, next, an electronic device according to another exemplary embodiment of the present invention is introduced.
[0156] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".
[0157] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for generating a hand data set according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 2 Steps 201-207 shown in .
[0158] Refer to the following Fig.12The electronic device 1200 according to this embodiment of the present invention is described. Fig.12 The electronic device 1200 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0159] like Fig.12 As shown, the electronic device 1200 is in the form of a general electronic device. The components of the electronic device 1200 may include but are not limited to: the at least one processor 1201, the at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).
[0160] Bus 1203 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0161] Computer storage media 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1221 and / or cache storage media 1222 , and may further include read-only computer storage media (ROM) 1223 .
[0162] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, such program modules 1224 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0163] The electronic device 1200 may also communicate with one or more external devices 1204 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1205. Furthermore, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1206. As shown, the network adapter 1206 communicates with other modules for the electronic device 1200 via a bus 1203. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0164] In some possible implementations, various aspects of a method for generating a hand dataset provided by the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to enable the computer device to execute the steps of the method for generating a hand dataset according to various exemplary embodiments of the present invention described above in this specification.
[0165] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for generating a hand data set, characterized in that: The method comprises: Using a semantic segmentation model to segment a real image containing a hand into a hand region, and obtaining at least one mask image of the hand region; Based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand area, the mask image of each hand area is screened to obtain the mask image of the target hand area; Using the mask image of the target hand region, identifying a hand boundary line of the target hand region, to obtain a mask image of the hand boundary line; For any background image in the initial hand data set, an initial hand composite image is obtained according to the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region; and Obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image; and Smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image; Based on the target hand synthetic image corresponding to each background image in the initial hand data set, a target hand data set is obtained.
2. The method according to claim 1, characterized in that The filtering of the mask images of the hand regions based on the 2D projection position coordinates of the hand key points in the real image and the mask images of the hand regions to obtain the mask image of the target hand region includes: For any one of the hand regions, based on the 2D projection position coordinates of multiple designated hand key points among the hand key points and the mask image of the hand region, determine whether the positional relationships between the multiple designated hand key points and the hand region all meet designated conditions, wherein the designated conditions include that the hand key point is inside the hand region, or that the hand key point is not inside the hand region and the edge distance between the hand key point and the hand region is less than a designated distance; If so, the hand region is determined to be the target hand region, and the mask image of the hand region is determined to be the mask image of the target hand region.
3. The method according to claim 1, characterized in that The step of identifying the hand boundary line of the target hand area by using the mask image of the target hand area to obtain the mask image of the hand boundary line includes: Performing morphological corrosion processing on the mask image of the target hand region to obtain the mask image of the target hand region after corrosion; and obtaining a mask image of the corroded hand boundary line in the target hand region based on the mask image after corrosion; and, Performing edge smoothing processing on the mask image of the target hand area, and performing edge detection on the smoothed mask image to obtain a smoothed mask image of the hand boundary line, and performing morphological dilation processing on the smoothed mask image of the hand boundary line to obtain an intermediate mask image of the hand boundary line; The mask image of the eroded hand boundary line and the intermediate mask image of the hand boundary line are fused to obtain the mask image of the hand boundary line.
4. The method according to claim 3, characterized in that The step of obtaining a mask image of the corroded hand boundary line in the target hand region based on the corroded mask image includes: For any pixel point in the mask image of the target hand area, subtract the pixel value of the pixel point in the mask image of the target hand area from the pixel value of the pixel point in the mask image after the corrosion, so as to obtain the target pixel value of the pixel point in the mask image of the corroded hand boundary line; According to the target pixel value of each pixel point, a mask image of the corroded hand boundary line is obtained.
5. The method according to claim 1, characterized in that The step of obtaining an initial hand composite image according to the background image and the hand region image comprises: Obtaining a target brightness of the hand area image according to a grayscale value of each pixel in the hand area image and a grayscale value of each pixel in the background image; The target brightness is used to adjust the brightness of the hand area image to obtain an adjusted hand area image; the adjusted hand area image is used as a foreground image to perform image fusion with the background image to obtain the initial hand composite image.
6. The method according to claim 5, characterized in that The step of obtaining the target brightness of the hand area image according to the grayscale value of each pixel in the hand area image and the grayscale value of each pixel in the background image comprises: Determine the median of the grayscale value of each pixel in the hand area image as the first intermediate brightness value; and Determine the median of the grayscale value of each pixel in the background image as the second intermediate brightness value; The target brightness is obtained according to the first middle brightness value and the second middle brightness value.
7. The method according to claim 6, characterized in that The step of obtaining the target brightness according to the first intermediate brightness value and the second intermediate brightness value includes: The second intermediate brightness value is divided by the first intermediate brightness value to obtain the target brightness.
8. The method according to claim 1, characterized in that The step of obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line includes: Using the mask image of the hand boundary line to interpolate the initial hand composite image to obtain an intermediate hand composite image; For any pixel point in the initial hand synthesis image, based on the pixel value of the pixel point in the initial hand synthesis image and the pixel value of the pixel point in the intermediate hand synthesis image, a pixel difference of the pixel point is obtained; and, If the pixel difference of the pixel point is greater than a specified threshold, the pixel point is determined as a boundary noise point; According to the position coordinates of each boundary noise point, the boundary noise point is marked in a preset binary image to obtain a mask image of the boundary noise point.
9. The method according to claim 1, characterized in that: The step of smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain the target hand composite image includes: The initial hand composite image is interpolated using the mask image of the boundary noise point to obtain the target hand composite image.
10. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Using a semantic segmentation model to segment a real image containing a hand into a hand region, and obtaining at least one mask image of the hand region; Based on the 2D projection position coordinates of each hand key point in the real image and the mask image of each hand area, the mask image of each hand area is screened to obtain the mask image of the target hand area; Using the mask image of the target hand region, identifying a hand boundary line of the target hand region, to obtain a mask image of the hand boundary line; For any background image in the initial hand data set, an initial hand composite image is obtained according to the background image and the hand region image, wherein the hand region image is obtained by segmenting the real image using the mask image of the target hand region; and Obtaining a mask image of boundary noise points in the initial hand composite image through the initial hand composite image and the mask image of the hand boundary line, wherein the boundary noise points are noise points located at the boundary of the initial hand composite image; and Smoothing the boundary noise points in the initial hand composite image according to the mask image of the boundary noise points to obtain a target hand composite image; Based on the target hand synthetic image corresponding to each background image in the initial hand data set, a target hand data set is obtained.