Method and system for generating gesture data set, and storage medium

Abstract

The present invention discloses a method for generating a gesture data set, a generating system, and a storage medium. The method for generating gesture data synthesis includes: using an infrared camera to collect an original image, wherein the original image includes gesture analysis data, gesture target data, and background data; preprocessing the original image to obtain segmented gesture analysis data, segmented gesture target data, and background data to be synthesized; adjusting the segmented gesture target data according to the segmented gesture analysis data to generate a target number of gesture target data to be synthesized; synthesizing the gesture target data to be synthesized and the background data to be synthesized to obtain an initial gesture synthesis image. The present invention generates a large number of different image data that are consistent with samples actually shot by the camera through means such as adjustment and fitting, thereby saving manpower and material resources for sample generation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for generating a gesture data set, and a storage medium. Background Art

[0002] In recent years, with the continuous development of deep learning theory and computer computing power, deep learning has made major breakthroughs in image recognition, image detection, image segmentation, generative adversarial networks and other fields in computer vision, and has been successfully applied in many fields such as face recognition, vehicle detection, face replacement and video restoration, and has achieved good results. However, due to the complexity of the task, the neural network model of deep learning usually contains millions of parameters. Training such a huge network model requires a large amount of sufficient data. For example, the famous ImageNet dataset contains more than 20,000 categories and about 15 million pictures. Therefore, large-scale labeled data is crucial for deep learning.

[0003] There are some deficiencies in the existing public gesture datasets, including: First, most of the data is RGB, while IR image data (IR image data is infrared image data output by infrared image sensors) is relatively small; second, data is usually collected based on the needs of the collector, which is unfriendly and inflexible for users other than the collector, and it is impossible to define the collection content according to the actual needs of the user, which is too restricted; third, there are differences in collection environment conditions, such as differences in collection cameras, which leads to differences in collected data, poor compatibility and inability to achieve good results; fourth, the data diversity is insufficient. For example, gestures at different positions and distances of the collector have great differences in details and brightness, and the application scenarios that can be covered are insufficient.

[0004] Although infrared images are increasingly used and generally accepted by people, actually collecting data is not an easy task. The task is too large and requires a lot of manpower, financial resources, time and energy, which makes it difficult to collect enough data. However, it is difficult to carry out subsequent work without new data sets, which will face a dilemma. Summary of the invention

[0005] In order to solve the technical problem in the prior art that it is difficult to obtain a sufficient number of gesture data sets, the present invention proposes a method for generating a gesture data set, a generating system, and a storage medium.

[0006] The method for generating gesture data synthesis proposed by the present invention includes:

[0007] Step 1, using an infrared camera to collect an original image, wherein the original image includes gesture analysis data, gesture target data and background data;

[0008] Step 2, preprocessing the original image to obtain segmented gesture analysis data, segmented gesture target data and background data to be synthesized;

[0009] Step 3, adjusting the segmented gesture target data according to the segmented gesture analysis data to generate a target number of gesture target data to be synthesized;

[0010] Step 4: synthesize the gesture target data to be synthesized and the background data to be synthesized to obtain an initial gesture synthesis image.

[0011] Furthermore, the method further includes: step 5, post-processing the initial synthesized image to obtain final gesture synthesis data.

[0012] Further, the infrared camera is used to collect a preset number of gesture images at different distances within an effective range, at different positions within the same field of view and with a blank background to form the gesture analysis data;

[0013] And / or using the infrared camera to collect a preset number of gesture images within a preset distance, at a central position within the same field of view, at different single angles and with a blank background to form the gesture target data;

[0014] And / or using the infrared camera to collect a preset number of images of various scenes without gestures within an effective range to form the background data.

[0015] Furthermore, the preprocessing includes performing data conversion on the gesture analysis data and gesture target data so that the pixel value of the original image satisfies the threshold interval [0, 255], and / or adjusting the size of the original image, and / or performing at least one of filtering, smoothing, and noise reduction processing on the original image.

[0016] Furthermore, the preprocessing includes performing one or a combination of operations of rotating, cropping, scaling, contrast adjustment, and brightness adjustment on the background data to obtain a target number of background images to be synthesized.

[0017] Further, the step 3 comprises:

[0018] Step 31, statistically analyzing the segmented gesture data to obtain a corresponding relationship between the mean and variance of the gesture data, and determining the value range of the mean and variance to form a reference benchmark condition;

[0019] Step 32, selecting the segmented gesture target data, and calculating the variance S1 and mean M1 of the gesture data to be adjusted;

[0020] Step 33, randomly selecting a mean M2 and variance S2 of a gesture data from the reference benchmark condition, adjusting the grayscale value of the pixel point of the gesture data to be adjusted point by point according to the formula H2=(H1-M1) / S1*S2+M2, and if the grayscale value of the pixel point after adjustment exceeds the threshold interval, assigning the grayscale value of the pixel point to the closest threshold value;

[0021] Step 34, adjusting the grayscale value of the pixel points of each segmented gesture target data image by repeating step 33 or step 32 and step 33, until the target number of gesture target images to be synthesized is reached.

[0022] Furthermore, the corresponding relationship between the mean and variance of the gesture data is:

[0023] max(0.28465*M+15.27581,0.1)≤S≤0.28465*M-25.27581, where S is the mean and M is the variance.

[0024] Furthermore, in step 4, during the synthesis of the initial gesture synthesis image, when the gesture target image to be synthesized is fitted, the gesture target image can form a closed gesture frame on the periphery, and the position of the gesture frame on the background image is represented by position L, which is the annotation information of the initial gesture synthesis image.

[0025] Further, the step 5 comprises:

[0026] The initial synthetic image is constructed as a matrix A, so that the number of rows and columns of the initial gesture synthetic image is equal to that of the matrix A; the maximum attenuation value at different locations in the field of view is randomly taken, and fitting is performed according to the distance relationship with the center of the circle, and the initial gesture cooperative image and the corresponding elements of the matrix A are multiplied one by one, so as to fit the gesture data at different positions in the lens as the final gesture synthetic data.

[0027] Furthermore, at least one of scattered point noise, straight line noise and curve noise or a combination thereof is randomly added to the gesture data at different positions in the fitted shot, and then used as the final gesture synthesis data.

[0028] The system for generating a gesture data set of the present invention adopts the method for generating a gesture data set described in the above technical solution to obtain gesture synthesis data, including:

[0029] An acquisition module, used for acquiring the original image;

[0030] A preprocessing module, for preprocessing the original image;

[0031] An adjustment module adjusts the processing result of the preprocessing module;

[0032] The synthesis module performs synthesis processing on the gesture target data to be synthesized and the background data to be synthesized.

[0033] The computer-readable storage medium of the present invention is used to store a computer program, which, when running, executes the method for generating a gesture data set described in the above technical solution to obtain gesture synthesis data.

[0034] The present invention can quickly and conveniently establish a gesture data set based on infrared camera acquisition without investing a large amount of human, material and financial resources. A small amount of IR image data is collected by the infrared camera, and gesture target data, background data, and the required basic reference conditions for adjustment are obtained after preprocessing, so as to quickly generate more gesture target data, and quickly generate more original gesture synthesis data with the background data, and then post-process the synthesized gesture data to be similar to the actual collected data, and automatically generate its annotation information, thereby avoiding the workload of annotation and providing more substantial data guarantee for deep learning. Compared with traditional methods, the efficiency of data collection and annotation is greatly improved, and the required training data set can be efficiently obtained with less investment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention is described in detail below with reference to the embodiments and accompanying drawings, wherein:

[0036] Figure 1 FIG. 1 is a flow chart of an embodiment of the present invention.

[0037] Figure 2 This is a relationship diagram between the mean and variance of a single gesture of the present invention.

[0038] Figure 3 This is a relationship diagram between the mean and variance of a large number of gestures of the present invention.

[0039] Figure 4 This is a representation of the bright surrounding pattern in the center of the IR image. DETAILED DESCRIPTION

[0040] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] Thus, a feature indicated in this specification will be used to illustrate one of the features of an embodiment of the present invention, rather than implying that each embodiment of the present invention must have the described feature. In addition, it should be noted that this specification describes many features. Although some features can be combined together to illustrate possible system designs, these features can also be used in other combinations that are not explicitly described. Thus, unless otherwise stated, the described combinations are not intended to be limiting.

[0042] The method for synthesizing gesture data sets of the present invention can quickly synthesize a large number of gesture images to form a new data set based on a small amount of collected IR gesture data, provide substantial training data when training a gesture classification network, and can also provide coordinate information of the gesture frame when training a gesture detection network, thereby greatly improving the diversity and collection efficiency of the training data set.

[0043] like Figure 1 As shown, the method for generating a gesture data set of the present invention includes at least 4 steps, and in a preferred embodiment, includes 5 steps.

[0044] In step 1 (S1), an infrared camera is used to collect an original image, and the original image includes gesture analysis data, gesture target data and background data.

[0045] The collection of gesture analysis data sets is to use infrared cameras to collect gesture images at different distances within the effective range, at different positions within the same field of view, and with a blank background, and to reach a certain number to form the original gesture analysis data set. That is, the infrared camera is used to collect a preset number of gesture images at different distances within the effective range, at different positions within the same field of view, and with a blank background to form gesture analysis data.

[0046] The collection of gesture target data sets is to use infrared cameras to collect gesture images within the effective range, at the center of the same field of view, at different single angles, and with a blank background, and to reach a certain number to form the original gesture target data set. That is, the infrared camera is used to collect a preset number of gesture images within a preset distance, at the center of the same field of view, at different single angles, and with a blank background to form gesture target data.

[0047] Collection of background data: Use infrared cameras to collect images of various scenes without gestures within the effective range, and reach a certain number to form the original background data set. That is, use infrared cameras to collect a preset number of images of various scenes without gestures within the effective range to form background data.

[0048] The acquisition order of the above original images is not specific.

[0049] Step 2 (S2) preprocesses the original image to obtain segmented gesture analysis data, segmented gesture target data, and background data to be synthesized. Specifically, the preprocessing includes performing data conversion on the gesture analysis data and the gesture target data so that the pixel value of the original image satisfies the threshold interval [0, 255], and / or adjusting the size of the original image, and / or performing at least one of filtering, smoothing, and noise reduction processing on the original image.

[0050] The gesture analysis data is preprocessed in the original gesture analysis image data. The data value of each pixel is different, which can be large or small, so the data type conversion is required. In this embodiment, the infrared camera used is a TOF camera. After statistics, it is found that the original data range is [0, 1023]. Since the pixel value of the original data range is set to exceed the value range [0, 255], it is not conducive to the subsequent calculation, and the original collected data needs to be normalized.

[0051] The normalization process includes: if the gray value of the collected pixel is P, the normalized gray value is set to P / 1023*255. Then the gesture frame is captured, and the image size is adjusted while ensuring that the aspect ratio of the gesture frame remains unchanged, and the length of the longest side is set to 180, so as to obtain a more uniform gesture analysis image.

[0052] Then, median filtering is performed to remove the extreme noise and smooth the gesture boundary. Then, the adaptive threshold method is used to segment the gesture parts. The largest connected domain is used as the valid gesture part, and the small connected domain is used as the removed noise. Then, the background is filled with a gray value of 255. If the gray value of a point in the valid gesture is 255, it is changed to 254, so the segmented gesture analysis data is obtained.

[0053] The steps and processes for preprocessing the gesture target data and the gesture analysis data are the same.

[0054] The purpose of preprocessing the background image is to adapt to more various scenes in reality. The background image can be rotated, cropped, scaled, contrasted, bright and darkened, etc., or adjusted in one or a combination thereof to obtain more background images to be synthesized.

[0055] In step 3 (S3), the segmented gesture target data is adjusted according to the segmented gesture analysis data to generate a target number of gesture target data to be synthesized, which specifically includes the following steps.

[0056] Step 31 , statistically analyzing the segmented gesture data to obtain a corresponding relationship between the mean and variance of the gesture data, and determining the value range of the mean and variance to form a reference benchmark condition.

[0057] In actual cameras, gestures at different distances have different grayscales. Gestures close to the camera are bright and have large changes in brightness, while gestures far from the camera are dark and have small changes in brightness. This situation of "low brightness and small changes for gestures far away, and high brightness and large changes for gestures close to the camera" can be used as a reference for adjusting gesture targets. Therefore, it is necessary to find the reference benchmark conditions corresponding to the adjustment of gesture targets. The specific method is to statistically analyze the segmented gesture analysis data and actually obtain a distribution and corresponding relationship of the gesture mean and variance (see Figure 2 and Figure 3 ), that is, the following relationship is satisfied:

[0058] max(0.28465*M+15.27581,0.1)≤S≤0.28465*M-25.27581

[0059] Among them, S is the mean and M is the variance.

[0060] from Figure 2 It can be found that if a single gesture is analyzed, the distribution of gesture mean and variance is more concentrated; Figure 3 In the data, it can be found that if multiple gestures are collected, the distribution of gesture mean and variance is more scattered; but it can be seen that the corresponding relationship between gesture mean and variance satisfies the above relationship, which shows that there is a high correlation between them. In other words, the gesture analysis data set is effective and can be used for the gesture target data set. Considering that the collection of the gesture analysis data set does not cover more real situations, we limit the gesture mean and standard deviation according to the actual situation; at the same time, in order to ensure that the training set can cover all more real situations, the mean M and standard deviation S of the training set can be appropriately relaxed. In this embodiment, the mean M and standard deviation S are respectively taken as: 45≤M≤225 and S≥0.1.

[0061] See Figure 3 The distribution of M is located on the horizontal axis of the figure, and the value of S is located on the vertical axis of the figure. However, in order to avoid the standard deviation being less than or equal to 0, the minimum value of S is 0.1. The data required for training should be distributed between Figure 3 Between the two slashes in .

[0062] Next, a segmented gesture target data (a segmented gesture target data image data) can be selected, and its mean M1 and variance S1 can be calculated; then a mean M2 and variance S2 are randomly selected from the reference benchmark conditions corresponding to the adjustment of the above gesture target, so that the following relationship is satisfied point by point for adjustment:

[0063] (H1-M1) / S1=(H2-M2) / S2

[0064] Among them, H1 is the original grayscale value of a pixel in an image, and H2 is the adjusted grayscale value.

[0065] That is, execute step 32 and step 33. Step 32 selects the segmented gesture target data, and calculates the variance S1 and mean M1 of the gesture data to be adjusted. Step 33 randomly selects the mean M2 and variance S2 of a gesture data from the reference benchmark condition, and adjusts the grayscale value of the pixel point of the gesture data to be adjusted point by point according to the formula H2 = (H1-M1) / S1*S2+M2. If the grayscale value of the adjusted pixel point exceeds the threshold interval, the grayscale value of the pixel point is assigned to the closest threshold. For example, after adjustment, if H2 < 0, then H2 = 0, if H2 > 254, then H2 = 254.

[0066] Step 34 adjusts the grayscale value of the pixel points of each segmented gesture target data image by repeating step 33 or step 32 and step 33 until the target number of gesture target images to be synthesized is reached. By repeatedly adjusting each segmented gesture target image, more and more gesture target images to be synthesized required in different scenarios can be obtained.

[0067] In step 4 (S4), the gesture target data to be synthesized and the background data to be synthesized are synthesized to obtain an initial gesture synthesis image. The specific method is to count the maximum and minimum values ​​of the side length of the gesture target image to be synthesized, randomly generate a number as the longest side within this interval, and then enlarge or reduce the gesture target image to be synthesized with the aspect ratio unchanged. Then, it is randomly rotated and / or mirrored; the part of the gesture target image whose data value is not 255 is pasted as the foreground to the background image to be synthesized, and the position L of the gesture frame is recorded at the same time, so as to obtain the initial gesture synthesis image. Based on the position L, the annotation information of this gesture in the initial gesture synthesis image is obtained, which can be used in subsequent deep learning. In some special cases, the initial gesture synthesis image can be used as a gesture-related data set with a large number of samples. Each synthesized image has been labeled and synchronized during the synthesis process. In step 4, when the initial gesture synthesis image (the fourth image set) is being synthesized, the gesture target image to be synthesized can form a closed circle on the periphery when it is fitted. This circle is called a gesture frame. The position of this frame (such as a rectangular frame to frame the gesture target) on the background image is specified and represented by position L. Position L is the annotation information of the initial gesture synthesis image.

[0068] In a preferred embodiment, the method may further include step 5 (S5), in which the initial synthesized image is post-processed to obtain final gesture synthesis data. Figure 4 As shown, under normal circumstances, the image captured by the camera will be brightest at the center of the image due to the lens, and then gradually darker and darker in concentric circles around it, which is also called Lens shading. The gesture target image to be synthesized will not exist in this situation, which will cause the initial synthesized image to be distorted, which is what the user does not want to see. In order to obtain an original image similar to or close to the original image captured by the camera, the initial synthesized image is constructed as a matrix A in step 5, so that the number of rows and columns of the initial gesture synthesized image and matrix A are equal; the maximum attenuation value at that location is randomly taken at different field of view angles (it can be understood as randomly taking a point or a random area. When processing images with different field of view angles, the maximum attenuation value is the maximum attenuation value of the point or the area in the currently selected field of view angle image), and fitting is performed according to the distance relationship with the center of the circle. The initial gesture cooperative image and the corresponding elements of matrix A are multiplied one by one, so as to fit the gesture data at different positions in the lens as the final gesture synthesis data. The gesture data after step 5 is the gesture data at different positions in the lens, which can meet more diverse needs. That is, according to the attenuation relationship of the lens from the center to the surrounding linear line, the attenuation gradually decreases from the center of the image to the surrounding areas. Then, the maximum attenuation value is randomly taken at different fields of view, and fitting is performed according to the distance relationship with the center of the circle. The initial gesture cooperation image and the corresponding elements of the matrix A are multiplied one by one, thereby fitting the gesture data at different positions in the lens.

[0069] In a further embodiment, at least one or a combination of scattered noise, linear noise and curve noise is randomly added to the gesture data at different positions in the fitted lens, and then used as the final gesture synthesis data. In this way, the obtained gesture data set includes sampling samples of more situations, which can basically meet all different data requirements for gestures. Since one image can be synthesized into multiple data, overfitting will easily occur, which is not conducive to the deep learning network to detect the gestures of different people, as well as the influence of objects such as rings and bracelets. In order to improve the robustness of the network, so that the network can better recognize the gestures of different people, and reduce the influence of objects such as rings and bracelets, it is necessary to increase noise. Several pixel points (U) can be randomly selected to add scattered noise, and the parameters such as the radius of the noise and whether the brightness decays from the center to the periphery can be randomly determined; then several straight lines (V) are randomly generated to add linear noise, and the radius, brightness and other parameters of the curve are randomly generated, and curve noise is added, and restored to a synthetic image with more noise through some external conditions.

[0070] After the above post-processing, the final gesture synthesis dataset is obtained. This dataset can be used for training. If it is used for gesture detection, this dataset and the corresponding position coordinates L recorded in step 4 can be used as the detection dataset. If it is used for gesture classification, the corresponding area can be cropped according to the position coordinates L recorded in step 4 as the classification dataset.

[0071] The present invention also protects a corresponding gesture data set generation system, which adopts the gesture data set generation method of the above technical solution to obtain gesture synthesis data, including:

[0072] An acquisition module, used for acquiring the original image;

[0073] A preprocessing module, for preprocessing the original image;

[0074] An adjustment module adjusts the processing result of the preprocessing module;

[0075] The synthesis module performs synthesis processing on the gesture target data to be synthesized and the background data to be synthesized.

[0076] The computer-readable storage medium of the present invention is used to store a computer program, which, when running, executes the method for generating a gesture data set described in the above technical solution to obtain gesture synthesis data.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a gesture data set, characterized in that: include: Step 1, using an infrared camera to collect an original image, wherein the original image includes gesture analysis data, gesture target data and background data; Using the infrared camera to collect gesture images at different distances within an effective range, at different positions within the same field of view and with a blank background to form the gesture analysis data; And / or using the infrared camera to collect gesture images within a preset distance, at a central position within the same field of view, at different single angles and with a blank background to form the gesture target data; and / or using the infrared camera to collect images of various scenes without gestures within an effective range to form the background data; Step 2, preprocessing the original image to obtain segmented gesture analysis data, segmented gesture target data and background data to be synthesized; Step 3, adjusting the segmented gesture target data according to the segmented gesture analysis data to generate a target number of gesture target data to be synthesized; Step 4, synthesizing the gesture target data to be synthesized and the background data to be synthesized to obtain an initial gesture synthesis image; The step 3 comprises: Step 31, performing statistics on the segmented gesture analysis data to obtain a corresponding relationship between the mean and variance of the gesture data, and determining the value range of the mean and variance to form a reference benchmark condition; Step 32, selecting the segmented gesture target data, and calculating the variance S1 and mean M1 of the gesture data to be adjusted; Step 33, randomly selecting a mean M2 and variance S2 of a gesture data from the reference benchmark condition, adjusting the grayscale value of the pixel point of the gesture data to be adjusted point by point according to the formula H2=(H1-M1) / S1*S2+M2, and if the grayscale value of the pixel point after adjustment exceeds the threshold interval, assigning the grayscale value of the pixel point to the closest threshold value; Step 34, adjusting the grayscale value of the pixel points of each segmented gesture target data image by repeating step 33 or step 32 and step 33 until the target number of gesture target images to be synthesized is reached; H1 is the original grayscale value of a pixel in an image, and H2 is the adjusted grayscale value.

2. The method for generating a gesture data set according to claim 1, characterized in that: Also includes: Step 5: post-process the initial synthesized image to obtain the final gesture synthesis data.

3. The method for generating a gesture data set according to claim 1, wherein: The preprocessing includes performing data conversion on the gesture analysis data and gesture target data so that the pixel value of the original image satisfies the threshold interval [0, 255], and / or adjusting the size of the original image, and / or performing at least one of filtering, smoothing, and noise reduction processing on the original image; and / or performing one or a combination of operations of rotating, cropping, scaling, contrast adjustment, and brightness adjustment on the background data to obtain a target number of background images to be synthesized.

4. The method for generating a gesture data set according to claim 1, wherein: The corresponding relationship between the mean and variance of the gesture data is max(0.28465*M+15.27581,0.1)≤S≤0.28465*M-25.27581, where S is the mean and M is the variance.

5. The method for generating a gesture data set according to claim 1, wherein: In step 4, during the synthesis of the initial gesture synthesis image, when the gesture target images to be synthesized are fitted, the gesture target images can form a closed gesture frame on the periphery, and the position of the gesture frame on the background image is represented by position L, which is the annotation information of the initial gesture synthesis image.

6. The method for generating a gesture data set according to claim 2, wherein: The step 5 comprises: The initial synthetic image is constructed into a matrix A, so that the number of rows and columns of the initial gesture synthetic image is equal to that of the matrix A; the maximum attenuation value at different locations in the field of view is randomly taken, and fitting is performed according to the distance relationship with the center of the circle, and the initial gesture cooperative image and the corresponding elements of the matrix A are multiplied one by one, so as to fit the gesture data at different locations in the lens; At least one of scattered point noise, straight line noise and curve noise or a combination thereof is randomly added to the gesture data at different positions in the fitted shot to obtain final gesture synthesis data.

7. A system for generating a gesture data set, characterized in that: The method for generating a gesture data set according to any one of claims 1 to 6 is used to obtain gesture synthesis data, comprising: An acquisition module, used for acquiring the original image; A preprocessing module, for preprocessing the original image; An adjustment module adjusts the processing result of the preprocessing module; The synthesis module performs synthesis processing on the gesture target data to be synthesized and the background data to be synthesized.

8. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is running, the method for generating a gesture data set as described in any one of claims 1 to 6 is executed to obtain gesture synthesis data.

Images