Image processing method, image processing device, storage medium and computer equipment

By obtaining grayscale and color gesture images at the same time, performing two-dimensional labeling and three-dimensional reconstruction, and automatically labeling grayscale gesture images, solving the problem of high efficiency and low cost of manual labeling in grayscale gesture recognition, and achieving efficient and convenient gesture labeling Dataset collection.

CN119068507AInactive Publication Date: 2024-12-03ZHUHAI MOJIE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310629647.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing grayscale camera gesture recognition technology requires a large amount of manual annotation data, which leads to high cost and low efficiency, making it difficult to efficiently and conveniently obtain gesture annotation data sets of grayscale images.

Method used

By obtaining the grayscale gesture images and color gesture images captured at the same time, performing two-dimensional key point annotation processing, reconstructing the three-dimensional coordinates, and projecting them to the grayscale gesture image, automatic marking of gesture key points is achieved and manual intervention is reduced.

Benefits of technology

It realizes efficient and convenient collection of gesture annotation data sets, reduces manual annotation costs, and improves labeling efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068507A_ABST
    Figure CN119068507A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method, an image processing device, a storage medium and computer equipment. The image processing method comprises the following steps: acquiring original gesture image data shot at the same moment, wherein the original gesture image data comprises a gray gesture image and at least two color gesture images with different shooting angles; performing two-dimensional key point labeling processing on each color gesture image to obtain a two-dimensional coordinate of each key point in each color gesture image; reconstructing according to the two-dimensional coordinates of each key point in all the color gesture images or according to the two-dimensional coordinates of each key point in part of the color gesture images to obtain three-dimensional coordinates of the key points; and projecting the three-dimensional coordinates of the key points to the gray gesture image to mark the key points in the gray gesture image to obtain a gesture key point mark image. According to the method, the collection process of the gesture labeling data set can be efficient and convenient, and manual labeling is not needed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular, relates to an image processing method, an image processing device, a computer-readable storage medium and a computer device. Background Art

[0002] With the development of machine vision and artificial intelligence technology in recent years, contactless gesture recognition, as one of the most natural ways of human-computer interaction, has broad application prospects. Most of the existing gesture recognition technologies are based on color cameras, while grayscale fisheye cameras have the advantages of large field of view and low power consumption, making them more suitable for mobile devices such as AR glasses. Gesture recognition technology based on grayscale cameras requires a large amount of annotated data for training. At present, the main method is to manually mark key points on grayscale images to form annotated data sets, which requires a lot of labor costs and is relatively inefficient. Summary of the invention

[0003] The technical problem solved by the present invention is: how to efficiently and conveniently obtain a gesture annotation dataset of a grayscale image.

[0004] The present application also discloses an image processing method, which includes:

[0005] Acquire original gesture image data captured at the same time, wherein the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles;

[0006] Performing two-dimensional key point labeling processing on each color gesture image to obtain the two-dimensional coordinates of each key point in each color gesture image;

[0007] Reconstruct the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in the entire color gesture image or according to the two-dimensional coordinates of each key point in a part of the color gesture image;

[0008] The three-dimensional coordinates of the key points are projected onto the grayscale gesture image to mark the key points in the grayscale gesture image, thereby obtaining a gesture key point marked image.

[0009] Optionally, the original gesture image data includes at least three color gesture images taken at different angles, and the method for reconstructing the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in part of the color gesture images includes:

[0010] Perform robustness analysis on the two-dimensional coordinates of each key point in each color gesture image, and select some two-dimensional coordinates that meet the labeling error conditions;

[0011] The three-dimensional coordinates of the key points are reconstructed based on the selected two-dimensional coordinates.

[0012] Optionally, the method for performing robustness analysis on the two-dimensional coordinates of each key point in each color gesture image includes:

[0013] At least two two-dimensional coordinates are selected from the two-dimensional coordinates of each key point in each color gesture image to reconstruct the initial three-dimensional coordinates of the key point;

[0014] Projecting the initial three-dimensional coordinates into each color gesture image to obtain the two-dimensional projection coordinates of the key point in each color gesture image;

[0015] The coordinate error value between the two-dimensional coordinates of the key point in each color gesture image and the two-dimensional projection coordinates is calculated, and the two-dimensional coordinates with a coordinate error value less than a threshold are taken as candidate two-dimensional coordinates to complete a robustness test.

[0016] Optionally, the method of screening out some two-dimensional coordinates that meet the annotation error condition includes:

[0017] Perform at least two robustness analyses on the two-dimensional coordinates of each key point in each color gesture image;

[0018] Count the number of candidate two-dimensional coordinates obtained from each robust analysis, and take each candidate two-dimensional coordinate obtained from the robust analysis with a number of candidate two-dimensional coordinates greater than a preset number as a partial two-dimensional coordinate that meets the labeling error condition; and / or count the average value of the coordinate error values ​​of the candidate two-dimensional coordinates obtained from each robust analysis, and take each candidate two-dimensional coordinate obtained from the robust analysis with an average value less than a preset average value as a partial two-dimensional coordinate that meets the labeling error condition.

[0019] Optionally, the method of reconstructing the three-dimensional coordinates of the key points according to the filtered part of the two-dimensional coordinates is:

[0020] The three-dimensional coordinates of the key points are obtained by performing triangulation calculation based on the selected two-dimensional coordinates.

[0021] Optionally, the image processing method further includes:

[0022] When the background of the gesture key point annotated image is a black screen background, segmenting the gesture area image in the gesture key point annotated image;

[0023] The gesture area image obtained by segmentation is pasted into a preset background image to obtain a gesture key point annotated image of the preset background.

[0024] Optionally, before acquiring the original gesture image data captured at the same time, the image processing method includes:

[0025] The gray-white fisheye camera and at least two color cameras are calibrated and synchronized, the gray-white fisheye camera is used to acquire the grayscale gesture image, and the at least two color cameras are used to acquire color gesture images from different shooting angles.

[0026] The present application also discloses an image processing device, which includes:

[0027] An image acquisition module, used to acquire original gesture image data captured at the same time, wherein the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles;

[0028] A labeling module is used to label each color gesture image with two-dimensional key points to obtain the two-dimensional coordinates of each key point in each color gesture image;

[0029] A reconstruction module, used to reconstruct the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in the entire color gesture image or according to the two-dimensional coordinates of each key point in part of the color gesture image;

[0030] The projection module is used to project the three-dimensional coordinates of the key points onto the grayscale gesture image to mark the key points in the grayscale gesture image to obtain a gesture key point marked image.

[0031] The present application also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program, and when the image processing program is executed by a processor, the above-mentioned image processing method is implemented.

[0032] The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and an image processing program stored in the computer-readable storage medium. When the image processing program is executed by the processor, the above-mentioned image processing method is implemented.

[0033] The present invention discloses an image processing method, an image processing device, a storage medium and a computer device, which have the following technical effects:

[0034] After obtaining the grayscale gesture image and the color gesture image at the same moment, this method performs two-dimensional annotation on the color gesture image, further reconstructs the three-dimensional coordinates of the key points, and finally projects the three-dimensional coordinates into the grayscale gesture image to obtain an annotated image of the gesture key points. The entire acquisition process does not require manual annotation, and the gesture acquisition process is efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0036] Figure 1 is a flowchart of an image processing method according to a first embodiment of the present invention;

[0037] Figure 2 is a principle block diagram of an image processing device according to a second embodiment of the present invention;

[0038] Figure 3 Schematic diagram of a computer device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Before describing the various embodiments of the present application in detail, the technical concept of the present application is briefly described first: in the prior art, manual annotation methods are often used to obtain gesture annotation datasets of grayscale images, which consumes huge labor costs and is inefficient. To this end, the image processing method provided by the present application performs key point annotation processing on the color gesture image taken at the same time as the grayscale gesture image to obtain two-dimensional coordinates, and further obtains the three-dimensional coordinates of the key points, and finally projects the three-dimensional coordinates to the grayscale gesture image to obtain the gesture key point annotation image. In this method, the collection process of the gesture annotation dataset is efficient and convenient, and no manual annotation is required, which improves the collection efficiency of gesture annotation data and reduces the cost of collecting gesture annotation data.

[0041] Specifically, Figure 1 As shown, the embodiment of the present application can be applied to a computer device, which can be a mobile phone, a computer, smart glasses, and a wearable device, etc., wherein the smart glasses can be AR (Augmented Reality) glasses or VR (Virtual Reality) glasses. The image processing method of the first embodiment includes the following steps:

[0042] Step S10, obtaining original gesture image data captured at the same time, wherein the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles;

[0043] Step S20, performing two-dimensional key point labeling processing on each color gesture image to obtain the two-dimensional coordinates of each key point in each color gesture image;

[0044] Step S30, reconstructing the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in the entire color gesture image or according to the two-dimensional coordinates of each key point in a part of the color gesture image;

[0045] Step S40: Projecting the three-dimensional coordinates of the key points onto the grayscale gesture image to mark the key points in the grayscale gesture image, thereby obtaining a gesture key point marked image.

[0046] Before step S10, it is necessary to build a gesture image acquisition platform to acquire grayscale gesture images and color gesture images. Among them, the gesture image acquisition platform is composed of a color camera and a gray-white fisheye camera. The gesture image acquisition platform can be set in a computer device, or it can be connected to the computer device for communication. For example, when the computer device is AR glasses, the color camera and the gray-white fisheye camera can be detachably connected to the AR glasses, or fixed in the AR glasses as a part of the AR glasses, or the color camera and the gray-white fisheye camera communicate with the AR glasses through Bluetooth, WIFI or wired connection to obtain grayscale gesture images and color gesture images. Among them, the number of color cameras is at least two, and gestures are collected at as many different angles as possible. On the one hand, the results of at least two color cameras can be used to use the triangulation method to perform three-dimensional reconstruction of a certain spatial point. At the same time, multiple color cameras are beneficial to ensure that there is a color camera that can directly capture the occluded part when the gesture is blocked and other difficult to identify situations, so as to obtain more accurate key point annotations, thereby reducing errors and making the measurement results more accurate. The number of gray-white fisheye cameras can also be set to multiple, and the shooting angles of each gray-white fisheye camera are different, so grayscale gesture images at different angles can be obtained. After the above steps are processed, gesture key point annotation images at multiple angles can be obtained. At the same time, the gestures to be collected will have different distortions at different positions of the gray-white fisheye camera. The use of multiple gray-white fisheye cameras can collect gesture image data at different angles and with different distortions, thereby improving the collection efficiency and data diversity. Therefore, the number of gray-white fisheye cameras used in this embodiment one is at least 1, and the number of color cameras is at least 2. The specific number can also be set according to actual needs. By way of example, the number of color cameras in this embodiment one is set to 6, and the number of gray-white fisheye cameras is set to 5.

[0047] Furthermore, in order to ensure that all cameras can capture the user's gestures during gesture collection, all color cameras and gray-white fisheye cameras need to be calibrated to determine the relationship between the three-dimensional position of a point in space and its corresponding point in each image. Exemplarily, the calibration method may adopt Zhang's calibration method. For example, assuming that the spatial coordinates of a point A in space in the world coordinate system are (U, V, W), when the camera is used to shoot, the pixel coordinates of point A in the image coordinate system are (u, v). In order to determine the transformation matrix M between the spatial coordinates (U, V, W) and the pixel coordinates (u, v), the camera needs to be calibrated, and a more mature calibration algorithm in the prior art may be used, such as Zhang's calibration method.

[0048] In order to capture the images of gestures at the same time point on each color camera and gray-white fisheye camera, it is necessary to synchronize all color cameras and gray-white fisheye cameras to avoid delays between cameras, so that gestures at the same time can be captured by each color camera and gray-white fisheye camera at the same time. For example, a common multi-camera synchronization method in the prior art can be used to synchronize the shooting of each color camera and gray-white fisheye camera. For example, the precise time synchronization protocol (PTP) is used to synchronize multiple cameras. Therefore, the precise time synchronization protocol can be used to synchronize each color camera and gray-white fisheye camera.

[0049] By continuously shooting gestures using a gesture image acquisition platform, grayscale gesture images and color gesture images at multiple moments can be obtained, thereby obtaining original gesture image data at multiple moments. It can be understood that at the same moment, there will be multiple grayscale gesture images and multiple color gesture images, and the original gesture image data includes grayscale gesture images and color gesture images. This embodiment describes the image processing method using the original gesture image data at the same moment as an example. It should be understood that after the original gesture image data at the remaining moments are processed by this method, the final gesture key point annotated image data set can be obtained. At the same time, it should be noted that each grayscale gesture image and color gesture image usually has multiple key points. For the convenience of description, this embodiment 1 uses a single key point as the object to illustrate the process of the image processing method, and the same image processing method is used for other key points.

[0050] Exemplarily, when performing two-dimensional key point annotation processing on the color gesture image in step S20, the mediapipe annotation algorithm can be used to perform two-dimensional key point annotation on the color gesture image, annotate the key points of the gesture in the color gesture image, and obtain the two-dimensional annotation point data of the key points, and the two-dimensional annotation point data of the key points include the two-dimensional coordinates of the key points in each color gesture image. The specific annotation process of the mediapipe annotation algorithm is well known to those skilled in the art and will not be described in detail here. In other embodiments, some common gesture key point detection models in the field of deep learning technology can also be used to perform two-dimensional key point annotation processing, for example, the hand key point detection model (hand pose estimation) under the deep learning framework OpenCV can be used, and the key point detection series model of the PaddleDetection model library can also be used. It should be noted that other algorithms that can realize two-dimensional annotation of gesture key points can also be applied in step S20 of the first embodiment of this invention.

[0051] Exemplarily, in step S30, the triangulation principle can be used to reconstruct the three-dimensional coordinates of the key point according to all or part of the two-dimensional coordinates corresponding to each key point. Among them, the number of the partial two-dimensional coordinates is at least two coordinates. For example, when there are only two color gesture images, that is, there are only two two-dimensional coordinates corresponding to the key point, the two two-dimensional coordinates are directly used to perform triangulation calculations to achieve three-dimensional reconstruction of the key point, and the three-dimensional coordinates of the key point can be obtained. When there are multiple color gesture images, for example, three or more, that is, when the two-dimensional coordinates corresponding to each key point are three or more, all the two-dimensional coordinates can be directly used to perform triangulation calculations to obtain the three-dimensional coordinates of the key point. It is also possible to select a part of the two-dimensional coordinates for reconstruction. Specifically, when the two-dimensional coordinates corresponding to each key point are three or more, a robustness analysis is performed on each two-dimensional coordinate, and the two-dimensional coordinates with large annotation errors are excluded, so that the annotation results are more authentic and reliable, so as to improve the spatial position accuracy of the reconstructed key points.

[0052] Exemplarily, the method of reconstructing the three-dimensional coordinates of each key point in the partial color gesture image according to the two-dimensional coordinates of the key point in step S30 includes the following steps:

[0053] Step S301, performing robustness analysis on the two-dimensional coordinates of each key point in each color gesture image, and screening out some two-dimensional coordinates that meet the labeling error condition;

[0054] Step S302: reconstruct the three-dimensional coordinates of the key points according to the selected two-dimensional coordinates.

[0055] The specific process of the robustness analysis in step S301 is: select at least two two-dimensional coordinates from the two-dimensional coordinates of each key point in each color gesture image to reconstruct the initial three-dimensional coordinates of the key point; project the initial three-dimensional coordinates into each color gesture image to obtain the two-dimensional projection coordinates of the key point in each color gesture image; calculate the coordinate error value between the two-dimensional coordinates of the key point in each color gesture image and the two-dimensional projection coordinates, and take the two-dimensional coordinates with a coordinate error value less than a threshold as candidate two-dimensional coordinates to complete a robustness test.

[0056] For example, after the initial three-dimensional coordinates (U1, V1, W1) of the key point are calculated by the triangulation method, the projection coordinates (u1, v1) corresponding to the initial three-dimensional coordinates (U1, V1, W1) in the color gesture image can be obtained according to the conversion matrix M obtained by the above calibration, thereby realizing projection. For each key point, there is a two-dimensional coordinate obtained by the two-dimensional key point annotation processing in step S20 and a two-dimensional projection coordinate obtained by projection in step S301 in the same color gesture image. In theory, the two-dimensional coordinates and the two-dimensional projection coordinates corresponding to the same key point will overlap, but due to the error of the two-dimensional key point annotation processing, the error of the camera calibration and other factors, the two-dimensional coordinates and the two-dimensional projection coordinates are usually not completely overlapped, that is, there is a coordinate error value between the two-dimensional coordinates and the two-dimensional projection coordinates. When there are multiple color gesture images, the two-dimensional coordinates, two-dimensional projection coordinates, and coordinate error values ​​corresponding to each key point are all multiple. The two-dimensional coordinates with a coordinate error value less than the threshold are taken as candidate two-dimensional coordinates, and the two-dimensional coordinates with a coordinate error value greater than or equal to the threshold are eliminated. Among them, the setting of the threshold is determined according to the actual situation. It is understandable that, under normal circumstances, each key point has a corresponding two-dimensional coordinate on each color gesture image, and each two-dimensional coordinate has a corresponding two-dimensional projection coordinate. The coordinate error value is the difference between the two-dimensional coordinate and the corresponding two-dimensional projection coordinate.

[0057] Furthermore, the method for selecting partial two-dimensional coordinates that meet the labeling error condition includes: performing at least two robustness analyses on the two-dimensional coordinates of each key point in each color gesture image; counting the number of candidate two-dimensional coordinates obtained by each robustness analysis, and taking each candidate two-dimensional coordinate obtained by the robustness analysis whose number of candidate two-dimensional coordinates is greater than a preset number as the partial two-dimensional coordinates that meet the labeling error condition; and / or counting the average value of the coordinate error values ​​of the candidate two-dimensional coordinates obtained by each robustness analysis, and taking each candidate two-dimensional coordinate obtained by the robustness analysis whose average value is less than a preset average value as the partial two-dimensional coordinates that meet the labeling error condition. The coordinate error value of the candidate two-dimensional coordinate is the coordinate error value between the two-dimensional coordinate selected as the candidate two-dimensional coordinate and the two-dimensional projection coordinate. For example, after one of the robustness analyses, a certain two-dimensional coordinate (u t , v t) is selected as a candidate two-dimensional coordinate, then the two-dimensional coordinate (u t , v t ) and the corresponding two-dimensional projection coordinates (u p , v p ) is directly used as the coordinate error value of the candidate two-dimensional coordinates.

[0058] For example, when there are n color gesture images captured at the same time, each key point has a corresponding two-dimensional coordinate in the n color gesture images, so there are n two-dimensional coordinates in total. When two two-dimensional coordinates are selected from the n two-dimensional coordinates each time to reconstruct the three-dimensional coordinates, and the three-dimensional coordinates are projected into the n color gesture images, n two-dimensional projection coordinates can be obtained, that is, n pairs of two-dimensional coordinates and two-dimensional projection coordinates can be obtained, and the two-dimensional coordinates and two-dimensional projection coordinates in each color gesture image are calculated to obtain the coordinate error value. After the error calculation is performed on the n color gesture images, n coordinate error values ​​can be obtained. Assuming that the number of coordinate error values ​​greater than the threshold value among the n coordinate error values ​​is m, and m is less than or equal to n, these m two-dimensional coordinates are all used as candidate two-dimensional coordinates, that is, the number of candidate two-dimensional coordinates is m.

[0059] Among them, when there are n color gesture images, each time two two-dimensional coordinates are selected, the robustness analysis can be repeated up to n*(n-1) / 2 times. Specifically, when the number of color gesture images in the first embodiment is 6, there are 6 two-dimensional coordinates corresponding to each key point. According to the permutation and combination rules, there are a total of 15 combinations of selecting two two-dimensional coordinates from the six two-dimensional coordinates, so 15 robustness analyses can be performed. It should be noted that the number of two-dimensional coordinates selected each time can be multiple, and the number of robustness analyses can also be multiple, which can be determined according to actual conditions.

[0060] Further, the screening criteria for the partial two-dimensional coordinates in the first embodiment are the number of candidate two-dimensional coordinates obtained after each robust analysis and the average value of the error coordinate values ​​of the candidate two-dimensional coordinates. It should be noted that when the screening criteria is the number of candidate two-dimensional coordinates, if the number of candidate two-dimensional coordinates obtained by multiple robust analyses is greater than the preset number, then the candidate two-dimensional coordinates obtained by one of the robust analyses can be arbitrarily selected as the partial two-dimensional coordinates, or the candidate two-dimensional coordinates obtained by the robust analysis with the largest number can be selected as the partial two-dimensional coordinates. If the number of candidate two-dimensional coordinates obtained by multiple robust analyses is greater than the preset number and the number of candidate two-dimensional coordinates of each time is the same, it can also be combined with the average value of the error coordinate value to screen, for example, the candidate two-dimensional coordinates obtained by one robust analysis with the smallest average value of the error coordinate value can be screened as the partial two-dimensional coordinates. Similarly, when the screening criteria is the average value of the error coordinate values ​​of the candidate two-dimensional coordinates, if the average value of the error coordinate values ​​obtained by multiple robust analyses is greater than the preset average value, then the candidate two-dimensional coordinates obtained by one of the robust analyses can be arbitrarily selected as the partial two-dimensional coordinates, or the candidate two-dimensional coordinates obtained by one robust analysis with the smallest average value can be selected as the partial two-dimensional coordinates. If the average value of the error coordinate values ​​obtained by multiple robustness analyses is less than the preset average value and the average values ​​of the error coordinate values ​​of the candidate two-dimensional coordinates of each time are the same, the number of candidate two-dimensional coordinates can also be combined for screening. For example, the candidate two-dimensional coordinates obtained by the robustness analysis with the largest number can be screened as part of the two-dimensional coordinates. The preset average value and the preset number can be set according to actual conditions.

[0061] Further, after obtaining the gesture key point annotated image in step S40, if the background of the gesture key point annotated image is a black screen background, the gesture area image in the gesture key point annotated image is segmented out, and the gesture area image is pasted into the preset background image to synthesize a new gesture key point annotated image. Among them, the background attributes of the color gesture image and the grayscale gesture image can be set in advance. For example, when the gesture to be collected by the experimental subject is in a pre-built black screen background, the background attributes of the color gesture image and the grayscale gesture image obtained by the color camera and the gray-white fisheye camera are set to the black screen background. When shooting under a non-black screen background such as a normal background, a white screen background or a blue screen background, the background attributes of the obtained color gesture image and the grayscale gesture image are set to a non-black screen background. The background attributes of the gesture key point annotated image and the grayscale gesture image are the same, and the background attributes can be used to determine whether the background of the gesture key point annotated image is a black screen background. The adaptive threshold segmentation method can be used to segment the gesture, and then it is pasted on the preset background image after rotation, changing the brightness and contrast, etc. When the background of the gesture key point annotated image is a non-black background, such as a normal background, segmentation processing is not required and it can be directly used to form a gesture key point annotated image set. The predetermined background can be different environmental scenes, different brightness scenes, different color backgrounds, etc. This can increase the richness and diversity of the gesture key point annotated image set and provide more abundant training samples for the training of the gesture recognition model.

[0062] Furthermore, if Figure 2 As shown, the image processing device of the second embodiment includes an image acquisition module 100, a labeling module 200, a reconstruction module 300 and a projection module 400. The image acquisition module 100 is used to acquire the original gesture image data captured at the same time, and the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles; the labeling module 200 is used to perform two-dimensional key point labeling processing on each color gesture image to obtain the two-dimensional coordinates of each key point in each color gesture image; the reconstruction module 300 is used to reconstruct the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in all color gesture images or according to the two-dimensional coordinates of each key point in part of the color gesture images; the projection module 400 is used to project the three-dimensional coordinates of the key point to the grayscale gesture image to label the key point in the grayscale gesture image to obtain a gesture key point labeling image.

[0063] Furthermore, the original gesture image data includes at least three color gesture images taken at different angles, and the reconstruction module 300 is also used to: perform robustness analysis on the two-dimensional coordinates of each key point in each color gesture image, screen out some two-dimensional coordinates that meet the labeling error conditions, and reconstruct the three-dimensional coordinates of the key point based on the screened partial two-dimensional coordinates.

[0064] Furthermore, the reconstruction module 300 is also used to: select at least two two-dimensional coordinates from the two-dimensional coordinates of each key point in each color gesture image to reconstruct the initial three-dimensional coordinates of the key point; project the initial three-dimensional coordinates into each color gesture image to obtain the two-dimensional projection coordinates of the key point in each color gesture image; calculate the coordinate error value between the two-dimensional coordinates of the key point in each color gesture image and the two-dimensional projection coordinates, and use the two-dimensional coordinates with a coordinate error value less than a threshold as candidate two-dimensional coordinates to complete a robustness analysis.

[0065] Furthermore, the reconstruction module 300 is also used to: perform at least two robustness analyses on the two-dimensional coordinates of each key point in each color gesture image; count the number of candidate two-dimensional coordinates obtained from each robustness analysis, and use each candidate two-dimensional coordinate obtained from the robustness analysis with a number of candidate two-dimensional coordinates greater than a preset number as a partial two-dimensional coordinate that meets the labeling error condition; and / or count the average value of the coordinate error values ​​of the candidate two-dimensional coordinates obtained from each robustness analysis, and use each candidate two-dimensional coordinate obtained from the robustness analysis with an average value less than a preset average value as a partial two-dimensional coordinate that meets the labeling error condition.

[0066] Furthermore, the reconstruction module 300 is also used to perform triangulation calculation according to each screened two-dimensional coordinate to obtain the three-dimensional coordinates of the key point.

[0067] Furthermore, the image processing device also includes a segmentation module 500 and a pasting module 600. The segmentation module 500 is used to segment out the gesture area image in the gesture key point annotation image of the black background, and the pasting module 600 is used to paste the segmented gesture area image into a preset background image to obtain a gesture key point annotation image of the preset background.

[0068] For more detailed processing procedures of each module in the image processing device, reference may be made to the relevant description of the first embodiment, which will not be described in detail here.

[0069] The third embodiment further discloses a computer-readable storage medium, which stores an image processing program. When the image processing program is executed by a processor, the image processing method in the first embodiment is implemented.

[0070] This fourth embodiment also discloses a computer device, at the hardware level, such as Figure 3As shown, the computer device includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and then runs it, forming a request processing device at the logical level. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logical unit, but can also be hardware or logic devices. An image processing program is stored on the computer-readable storage medium 11, and the image processing program implements the above-mentioned image processing method when executed by the processor.

[0071] Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0072] The specific implementation methods of the present invention are described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that these embodiments can be modified and improved without departing from the principles and spirit of the present invention, the scope of which is defined by the claims and their equivalents. These modifications and improvements should also be within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquire original gesture image data captured at the same time, wherein the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles; Performing two-dimensional key point labeling processing on each color gesture image to obtain the two-dimensional coordinates of each key point in each color gesture image; Reconstruct the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in the entire color gesture image or according to the two-dimensional coordinates of each key point in a part of the color gesture image; The three-dimensional coordinates of the key points are projected onto the grayscale gesture image to mark the key points in the grayscale gesture image, thereby obtaining a gesture key point marked image.

2. The image processing method according to claim 1, characterized in that: The original gesture image data includes at least three color gesture images shot at different angles, and the method for reconstructing the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in part of the color gesture image includes: Perform robustness analysis on the two-dimensional coordinates of each key point in each color gesture image, and select some two-dimensional coordinates that meet the labeling error conditions; The three-dimensional coordinates of the key points are reconstructed based on the selected two-dimensional coordinates.

3. The image processing method according to claim 2, characterized in that: The method for performing robustness analysis on the two-dimensional coordinates of each key point in each color gesture image includes: At least two two-dimensional coordinates are selected from the two-dimensional coordinates of each key point in each color gesture image to reconstruct the initial three-dimensional coordinates of the key point; Projecting the initial three-dimensional coordinates into each color gesture image to obtain the two-dimensional projection coordinates of the key point in each color gesture image; The coordinate error value between the two-dimensional coordinates of the key point in each color gesture image and the two-dimensional projection coordinates is calculated, and the two-dimensional coordinates with a coordinate error value less than a threshold are taken as candidate two-dimensional coordinates to complete a robustness analysis.

4. The image processing method according to claim 3, characterized in that: The method for screening out some two-dimensional coordinates that meet the annotation error condition includes: Perform at least two robustness analyses on the two-dimensional coordinates of each key point in each color gesture image; Count the number of candidate two-dimensional coordinates obtained from each robust analysis, and take each candidate two-dimensional coordinate obtained from the robust analysis with a number of candidate two-dimensional coordinates greater than a preset number as a partial two-dimensional coordinate that meets the labeling error condition; and / or count the average value of the coordinate error values ​​of the candidate two-dimensional coordinates obtained from each robust analysis, and take each candidate two-dimensional coordinate obtained from the robust analysis with an average value less than a preset average value as a partial two-dimensional coordinate that meets the labeling error condition.

5. The image processing method according to claim 2, characterized in that: The method of reconstructing the three-dimensional coordinates of the key points based on the selected two-dimensional coordinates is: The three-dimensional coordinates of the key points are obtained by performing triangulation calculation based on the selected two-dimensional coordinates.

6. The image processing method according to claim 1, characterized in that: The image processing method further comprises: When the background of the gesture key point annotated image is a black screen background, segmenting the gesture area image in the gesture key point annotated image; The gesture area image obtained by segmentation is pasted into a preset background image to obtain a gesture key point annotated image of the preset background.

7. The image processing method according to claim 1, characterized in that: Before acquiring the original gesture image data captured at the same time, the image processing method includes: The gray-white fisheye camera and at least two color cameras are calibrated and synchronized, the gray-white fisheye camera is used to acquire the grayscale gesture image, and the at least two color cameras are used to acquire color gesture images from different shooting angles.

8. An image processing device, characterized in that: The image processing comprises: An image acquisition module, used to acquire original gesture image data captured at the same time, wherein the original gesture image data includes a grayscale gesture image and at least two color gesture images captured at different angles; A labeling module is used to label each color gesture image with two-dimensional key points to obtain the two-dimensional coordinates of each key point in each color gesture image; A reconstruction module, used to reconstruct the three-dimensional coordinates of each key point according to the two-dimensional coordinates of each key point in the entire color gesture image or according to the two-dimensional coordinates of each key point in part of the color gesture image; The projection module is used to project the three-dimensional coordinates of the key points onto the grayscale gesture image to mark the key points in the grayscale gesture image to obtain a gesture key point marked image.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image processing program, and when the image processing program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The computer device includes a computer-readable storage medium, a processor, and an image processing program stored in the computer-readable storage medium. When the image processing program is executed by the processor, the image processing method according to any one of claims 1 to 7 is implemented.