A method, apparatus, and electronic device for skeletal key point correction

Abstract

This application discloses a method, apparatus, and electronic device for correcting skeletal key points. The method includes: acquiring an RGB image and a depth image of a human body; performing human body recognition on the RGB image to obtain human skeletal key points; determining whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal key points in the depth image; if pixels with abrupt changes in depth value exist, delineating a local region based on adjacent skeletal key points, traversing pixels within the local region, and determining the depth value difference between the traversed pixel and its adjacent pixels; determining the boundary pixels belonging to the human body contour based on the traversal results, and marking the corresponding boundary pixels in the RGB image; and re-performing human body recognition on the marked RGB image to obtain the corrected skeletal key points of the human body. This application first evaluates the positional accuracy of the key skeletal points, and then finely processes the local RGB image for correction, resulting in fast calculation speed and good correction effect.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, and electronic device for correcting skeletal key points. Background Technology

[0002] Deep learning algorithms (such as AlphaPose) can be used to detect two-dimensional skeletal key points of the human body from RGB images, including skeletal key points of the head, torso, and limbs. Moreover, a significant number of scenario-based applications of skeletal key points are tailored to specific needs based on the posture and movements of the human limbs.

[0003] However, various reasons often lead to inaccurate or off-target detection of key skeletal points in the limbs (such as elbows, knees, wrists, and ankles). These reasons include external factors such as excessively dark shooting environments, low resolution of the RGB camera, and similarity in color or texture between the human body and the background. Figure 1 As shown; internal factors such as inherent defects in the algorithm itself, or defects in the training data of the deep learning model, etc.

[0004] The existing technical solution involves obtaining the overall outline of the human body from a depth camera, modifying the RGB image accordingly, and then obtaining the skeletal key points. This solution requires obtaining the entire human body outline, which is computationally intensive and lacks precision. Therefore, a solution is needed that can correct for inaccurate or misaligned skeletal key point detection. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, and electronic device for correcting skeletal key points, in order to solve the problem that existing correction methods are computationally intensive and lack precision when skeletal key point detection is inaccurate or deviates from the actual position.

[0006] In a first aspect, embodiments of this application provide a method for correcting skeletal key points, the method comprising:

[0007] Acquire RGB and depth images of the human body, and perform human body recognition on the RGB images to obtain the key skeletal points of the human body;

[0008] Based on the depth values ​​of pixels between adjacent skeletal key points in the depth image, determine whether there are pixels with abrupt changes in depth values;

[0009] If there are pixels with a sudden change in depth value, a local region is defined based on the adjacent skeletal key points. The pixels in the local region are traversed to determine the traversed pixels and the adjacent pixels adjacent to the traversed pixels, and the depth value difference between the traversed pixels and the adjacent pixels is determined.

[0010] Based on the traversal results, the boundary pixels belonging to the human body contour are determined, and the corresponding boundary pixels are marked in the RGB image.

[0011] Human body recognition is performed again on the annotated RGB image to obtain the corrected skeletal key points of the human body.

[0012] In one or more possible embodiments, determining whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal keypoints in the depth image includes:

[0013] By connecting adjacent skeletal key points in the depth image as endpoints, multiple pixels located on the connecting lines in the depth image are determined.

[0014] For any intermediate pixel p among the plurality of pixels, excluding the first and last pixels... i Calculate the mean d of the absolute values ​​of the depth differences between the middle pixel and its two adjacent pixels. Δi ;

[0015] Determine the mean d of all intermediate pixels. Δi mean d s If the mean d s Greater than the preset threshold t s At that time, it was determined that there were pixels where the depth value changed abruptly.

[0016] In one or more possible embodiments, if there are pixels where the depth value changes abruptly, the method further includes:

[0017] Identify the location of the depth value abrupt change corresponding to the pixel where the depth value abruptly changes;

[0018] Centered on the depth value abrupt change location, calculate the depth value difference between the depth value abrupt change location and the surrounding neighboring pixels, and determine the maximum absolute value of the depth value difference as the depth value difference threshold.

[0019] The step of determining the boundary pixels belonging to the human body contour based on the traversal results includes:

[0020] When the absolute value of the depth difference between a traversed pixel and any neighboring pixel exceeds the depth difference threshold, the traversed pixel is determined to be a boundary pixel of the human body contour.

[0021] In one or more possible embodiments, determining the depth value abrupt change location corresponding to the pixel where the depth value abrupt change occurs includes:

[0022] Determine the mean d of all intermediate pixels. Δi The middle pixel corresponding to the maximum value is the location of the depth value change corresponding to the pixel where the depth value changes abruptly.

[0023] In one or more possible embodiments, defining a local region based on the adjacent skeletal keypoints includes:

[0024] Determine the distance L between the adjacent skeletal key points;

[0025] Using the inaccurately identified bone key points among the adjacent bone key points as the center, draw a circle with a radius of w times the distance L, and determine the area composed of all pixels within the circle as the local region.

[0026] In one or more possible embodiments, marking corresponding boundary pixels in the RGB image includes:

[0027] Determine the corresponding boundary pixels in the RGB image;

[0028] The color values ​​of the boundary pixels in the RGB image are set to their complementary colors to obtain the annotated RGB image.

[0029] In one or more possible embodiments, the adjacent skeletal key points include a first group of skeletal key points consisting of shoulder and elbow skeletal key points, a second group of skeletal key points consisting of elbow and wrist skeletal key points, a third group of skeletal key points consisting of hip and knee skeletal key points, and a fourth group of skeletal key points consisting of knee and ankle skeletal key points, wherein:

[0030] The first and third sets of skeletal key points are identified sequentially, followed by the second and fourth sets of skeletal key points.

[0031] If there are no pixels with a sudden change in depth value in the first and third groups of bone keypoints being traversed, traverse the second and fourth groups of bone keypoints. If there are pixels with a sudden change in depth value in the first or third group of bone keypoints being traversed, annotate the RGB image once, and then traverse the second and fourth groups of bone keypoints.

[0032] After completing the traversal of all groups of skeletal key points, trigger the re-human recognition of the currently annotated RGB image.

[0033] In one or more possible embodiments, the adjacent skeletal key points include a first group of skeletal key points consisting of shoulder and elbow skeletal key points, a second group of skeletal key points consisting of elbow and wrist skeletal key points, a third group of skeletal key points consisting of hip and knee skeletal key points, and a fourth group of skeletal key points consisting of knee and ankle skeletal key points, wherein:

[0034] The first and third sets of skeletal key points are identified sequentially, and then the second and fourth sets of skeletal key points are identified.

[0035] If there are no pixels with abrupt changes in depth value in the first and third groups of skeletal keypoints during the current traversal, traverse the second and fourth groups of skeletal keypoints. Otherwise, annotate the RGB image once, perform human recognition on the annotated RGB image to correct the first or third group of skeletal keypoints with recognition errors, and then traverse the second and fourth groups of skeletal keypoints after correction.

[0036] If there are pixels with abrupt changes in depth value in the second and fourth sets of skeletal keypoints during the current traversal, annotate the RGB image once, and perform human recognition on the annotated RGB image to correct the second or fourth set of skeletal keypoints with recognition errors.

[0037] In one or more possible embodiments, calculating the depth value difference between the depth value abrupt change region and its surrounding neighboring pixels, centered on the depth value abrupt change region, includes:

[0038] Centered on the location of the abrupt change in depth value, calculate the depth value difference between the location of the abrupt change and the four surrounding neighboring pixels; or

[0039] Centered on the depth value abrupt change location, calculate the depth value difference between the depth value abrupt change location and the surrounding eight neighboring pixels.

[0040] Secondly, this application provides a device for correcting skeletal key points, the device comprising:

[0041] The image acquisition module is used to acquire RGB images and depth images of the human body, and to perform human body recognition on the RGB images to obtain the key skeletal points of the human body.

[0042] The anomaly determination module is used to determine whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal key points in the depth image.

[0043] The depth value traversal module is used to, if there are pixels with a sudden change in depth value, delineate a local region based on the adjacent skeletal key points, traverse the pixels in the local region, determine the traversed pixels and the adjacent pixels adjacent to the traversed pixels, and determine the depth value difference between the traversed pixels and the adjacent pixels.

[0044] The boundary determination module is used to determine the boundary pixels belonging to the human body contour based on the traversal results, and to mark the corresponding boundary pixels in the RGB image.

[0045] The key point correction module is used to re-identify the human body in the annotated RGB image to obtain the corrected skeletal key points of the human body.

[0046] Thirdly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the skeletal keypoint correction methods provided in the embodiments of this application.

[0047] Fourthly, another embodiment of this application also provides a computer storage medium storing a computer program for causing a computer to execute any of the skeletal keypoint correction methods provided in the embodiments of this application.

[0048] In this embodiment, two-dimensional skeletal points obtained from any human skeletal keypoint detection algorithm are processed. The two-dimensional skeletal points are judged from the corresponding depth image to determine whether there are errors. For erroneous skeletal points, local areas are delineated, the RGB images of the corresponding areas are modified, and the positions of these skeletal points are regenerated. The computational load is small and the correction result is relatively accurate, thereby solving the problems of existing skeletal keypoint correction methods, which have large processing load, long processing time, and inaccurate results.

[0049] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is an illustration illustrating how the human body, when its color or texture is similar to the background, can easily lead to misidentification of skeletal features, as an example of this application.

[0052] Figure 2 This is a schematic diagram of the application environment of an embodiment of this application;

[0053] Figure 3 This is a schematic flowchart of a method for correcting skeletal key points provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of the overall process of the skeletal key point correction method provided in the embodiments of this application;

[0055] Figure 5 This is a flowchart illustrating the process of determining whether a pixel has undergone a sudden change in depth value in this embodiment;

[0056] Figure 6 This is a schematic diagram illustrating the process of determining the depth difference of all intermediate pixels in an embodiment of this application;

[0057] Figure 7 This is a flowchart illustrating the process of determining whether the location of key points of the elbow (knee) bones is incorrect, as exemplified in the embodiments of this application.

[0058] Figure 8 This is a schematic diagram illustrating an error in identifying key points of the elbow (knee) bones, as exemplified in an embodiment of this application.

[0059] Figure 9 This is a schematic diagram illustrating a wrist (ankle) bone key point recognition error as an example of an embodiment of this application;

[0060] Figure 10 This is a schematic diagram illustrating the process of determining the depth difference threshold for boundary determination in an embodiment of this application;

[0061] Figure 11 This is an example of determining a depth difference threshold based on the depth difference between a determined depth value abrupt change location and its surrounding neighboring pixels in an embodiment of this application.

[0062] Figure 12 This is a flowchart illustrating the process of determining the boundary pixels of the human body contour within a defined local area in an embodiment of this application.

[0063] Figure 13 This is a flowchart illustrating the process of determining the location of a sudden change in depth value in an embodiment of this application;

[0064] Figure 14 This is a schematic diagram of the process for defining a local area in an embodiment of this application;

[0065] Figure 15 This is a schematic diagram illustrating the process of annotating RGB images using boundary pixels in an embodiment of this application;

[0066] Figure 16 This is a schematic diagram of an RGB image showing the marked boundary pixels, as exemplified in an embodiment of this application.

[0067] Figure 17 This is a schematic diagram of a skeletal key point correction device provided in an embodiment of this application;

[0068] Figure 18This is a schematic diagram of another skeletal key point correction device provided in the embodiments of this application;

[0069] Figure 19 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0070] In the description of the embodiments of this application, unless otherwise stated, the term "multiple" refers to two or more, and other quantifiers are similarly understood. The preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0071] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0072] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0073] See Figure 2 This is a schematic diagram of an application scenario according to an embodiment of this application.

[0074] like Figure 2 As shown, in this application environment, for example, it can be as follows: Figure 1 As shown, this application environment may include, for example, a network 10, a server 20, at least one camera device 30, a terminal device 40, and a database 50. Specifically, the camera device 30 is used to capture images of human bodies within the monitoring range and transmits these images to the server 20 via the network 10, whereby the server 20 stores the images in the database 50.

[0075] The human images captured by the camera device 30 can be sent to the server 20 via the network 10, and the server 20 can issue a storage instruction to store them in the database 50 associated with the server 20.

[0076] Of course, the terminal device 40 can also integrate a camera device to directly capture human images through the local camera device. The terminal device 40 can then perform human recognition on the human images it has captured or the server can perform human image recognition on the human images captured from the database to obtain the key points of the human skeleton. Alternatively, the terminal device 40 can integrate a camera device to directly capture human images through the local camera device, and then send the human images to the server. The server will then process the human images to obtain the human skeleton points.

[0077] The server processes the obtained human skeletal point information and sends it back to the terminal device, or the server further processes the obtained human skeletal points and then feeds back the relevant information to the terminal device.

[0078] In view of the problems of inaccurate or non-precise correction methods for skeletal keypoint detection in related technologies, such as large computational load and low precision, this application provides a method for skeletal keypoint correction. Based on RGB and depth images acquired by a depth camera, it can detect and correct two-dimensional skeletal keypoints with inaccurate position detection results caused by factors such as the consistency between the environmental background and the human clothing color tone. Figure 3 As shown, the method includes:

[0079] Step 301: Obtain the RGB image and depth image of the human body, and perform human body recognition on the RGB image to obtain the key points of the human skeleton;

[0080] This application embodiment can use a depth camera to simultaneously acquire RGB images and depth images. For the RGB images, deep learning algorithms can be used for human body recognition to obtain the key points of the human skeleton. Based on the depth images, the depth value information of each pixel can be obtained.

[0081] Step 302: Based on the depth values ​​of pixels between adjacent skeletal key points in the depth image, determine whether there are pixels with abrupt changes in depth values;

[0082] There are multiple pixels between adjacent skeletal keypoints, and each pixel has a corresponding depth value. By comparing the depth values ​​of pixels between adjacent skeletal keypoints, it is determined whether there are any pixels with abrupt changes in depth value.

[0083] Step 303: If there are pixels with a sudden change in depth value, a local region is defined based on the adjacent skeletal key points, pixels in the local region are traversed, the traversed pixels and the adjacent pixels adjacent to the traversed pixels are determined, and the depth value difference between the traversed pixels and the adjacent pixels is determined.

[0084] The depth values ​​between adjacent skeletal keypoints located within the human body contour should be relatively similar. If there are pixels with abrupt changes in depth value, it indicates that one of the skeletal keypoints is not located on the human body contour, resulting in inaccurate recognition. In this embodiment, when the skeletal keypoint recognition result is determined to be inaccurate, a local region is defined based on adjacent skeletal keypoints. This local region includes the boundary of the human body contour and contains the correct skeletal keypoints that correct the incorrect ones. To identify the human body contour boundary, the pixels within this local region are traversed, and the depth value difference between the traversed pixels and their adjacent pixels is determined.

[0085] Step 304: Determine the boundary pixels belonging to the human body contour based on the traversal results, and mark the corresponding boundary pixels in the RGB image;

[0086] Since the depth difference between the boundary pixels of the human body contour and the surrounding pixels outside the human body contour is large, the boundary pixels belonging to the human body contour can be filtered out by traversing the depth difference between the pixels and the adjacent pixels, and the corresponding boundary pixels can be marked in the RGB image. The specific marking method can be to change their color value or other methods.

[0087] Step 305: Perform human body recognition again on the annotated RGB image to obtain the corrected skeletal key points of the human body.

[0088] The skeletal keypoint correction method provided in this application obtains the skeletal keypoints of the human body from an RGB image, then identifies erroneous skeletal keypoints based on a depth image, then delineates a local region and modifies the RGB image in that region, and finally regenerates the erroneous skeletal keypoints based on the modified RGB image, thereby achieving the correction of the skeletal keypoints. This application first evaluates the positional accuracy of the key skeletal points, and then finely processes the local RGB image to achieve the effect of correcting the position of the skeletal points. The calculation speed is fast and the correction effect is good.

[0089] In the embodiments of this application, adjacent skeletal key points include a first group of skeletal key points composed of shoulder skeletal key points and elbow skeletal key points, a second group of skeletal key points composed of elbow skeletal key points and wrist skeletal key points, a third group of skeletal key points composed of hip skeletal key points and knee skeletal key points, and a fourth group of skeletal key points composed of knee skeletal key points and ankle skeletal key points. Among the aforementioned skeletal key points, the shoulder and hip skeletal key points are trunk skeletal key points. According to relevant skeletal key point detection algorithms, trunk skeletal key points are identified by combining information such as trunk contours. Furthermore, trunk skeletal key points have relatively small ranges of motion, thus the identification results are relatively accurate. In this embodiment, trunk skeletal key points are assumed to be accurately identified. Then, following the direction from the trunk to the limbs, a set of skeletal key points is determined sequentially. At least one set of skeletal key points is determined for each traversal, following the direction from the trunk to the limbs. If an inaccurately identified set of skeletal key points is found during the current traversal, since there are pre-determined accurately identified skeletal key points in that set, the other inaccurately identified skeletal key point is corrected. In the next traversal, the corrected skeletal key point is used as the accurately identified skeletal key point. Specifically, the correction of all sets of skeletal key points can be achieved in the following way:

[0090] The first and third sets of skeletal key points are identified sequentially, and then the second and fourth sets of skeletal key points are identified.

[0091] If there are no pixels with abrupt changes in depth value in the first and third groups of skeletal keypoints during the current traversal, traverse the second and fourth groups of skeletal keypoints. Otherwise, annotate the RGB image once, perform human recognition on the annotated RGB image to correct the first or third group of skeletal keypoints with recognition errors, and then traverse the second and fourth groups of skeletal keypoints after correction.

[0092] If there are pixels with abrupt changes in depth value in the second and fourth sets of skeletal keypoints during the current traversal, annotate the RGB image once, and perform human recognition on the annotated RGB image to correct the second or fourth set of skeletal keypoints with recognition errors.

[0093] In this embodiment, the trunk skeletal key points are assumed to be the accurate skeletal key points. If there are pixels with a sudden change in depth value in the first group of skeletal key points / the third group of skeletal key points being traversed, it is determined that the elbow skeletal key points / knee skeletal key points are not accurately identified. Then, the elbow skeletal key points / knee skeletal key points are corrected to the correct skeletal key points by annotating the RGB image and re-human body.

[0094] Since the elbow / knee bone key points are corrected to the correct bone key points during the second traversal, if there are pixels with abrupt changes in depth value in the second / fourth group of bone key points during the current traversal, it is determined that the wrist / ankle bone key points are not accurately identified.

[0095] In this embodiment, the identification and correction of all groups of skeletal key points are completed through two traversals. Steps 302 to 305 are executed once during each traversal. The specific process is as follows: Figure 4 As shown, it includes:

[0096] Step 401: Obtain the RGB image and depth image of the human body, and perform human body recognition on the RGB image to obtain the key points of the human skeleton;

[0097] Step 402: Perform the traversal in the order of first identifying the first group of skeletal key points and the third group of skeletal key points, and then identifying the second group of skeletal key points and the fourth group of skeletal key points.

[0098] Step 403: For each group of skeletal key points currently traversed, based on the depth values ​​of the pixels between the skeletal key points in the depth image, determine whether there are any pixels with abrupt changes in depth values. If there are pixels with abrupt changes in depth values, proceed to step 404; otherwise, return to step 402.

[0099] Step 404: If there are pixels with a sudden change in depth value, define a local region based on the set of skeletal key points, traverse the pixels in the local region, and determine the depth value difference between the traversed pixels and their adjacent pixels.

[0100] Step 405: Determine the boundary pixels belonging to the human body contour based on the traversal results, and mark the corresponding boundary pixels in the RGB image;

[0101] Step 406: Re-perform human body recognition on the labeled RGB image and correct any inaccurate skeletal key points encountered during the current iteration.

[0102] Step 407: Determine if the traversal is complete. If not, return to step 402. If the traversal is complete, then end.

[0103] In one or more possible embodiments, based on the depth values ​​of pixels between adjacent skeletal keypoints in the depth image, it is determined whether there are pixels with abrupt changes in depth values, such as... Figure 5 As shown, it includes:

[0104] Step 501: Connect adjacent skeletal key points in the depth image with endpoints to determine multiple pixels located on the connecting lines in the depth image;

[0105] Taking the first or third set of skeletal key points as an example, take the shoulder (hip) skeletal key point a and the elbow (knee) skeletal key point b identified by the algorithm. F Let all n pixels on the line connecting them be denoted as p1, p2, ..., pn. n Each point has an x-coordinate, y-coordinate, RGB value, and depth value, denoted as d1, d2, ..., dn. n For any pixel in the depth image, its RGB value is determined by the correspondence between the depth image and RGB, which is the RGB value of the pixel in the RGB image.

[0106] Step 502, for any intermediate pixel p among the plurality of pixels excluding the first and last pixels. i Calculate the mean d of the absolute values ​​of the depth differences between the middle pixel and its two adjacent pixels. Δi ;

[0107] Specifically, such as Figure 6 As shown, it mainly includes:

[0108] Step 601: Extract the key points a of the shoulder (hip) bones identified by the algorithm and the key points b of the elbow (knee) bones identified by the algorithm. F Let all n pixels on the line connecting them be denoted as p1, p2, ..., pn. n ;

[0109] Step 602, obtain p2, p3, ..., p n-1 Any point p among these n-2 points i ;

[0110] Step 603, calculate its relationship with the two adjacent pixels p. i-1 and p i+1 The mean of the absolute values ​​of the depth differences d Δi ,Right now:

[0111] d Δi =(|d i -d i-1 |+|d i -d i+1 |) / 2;

[0112] Step 604, calculate p2, p3, ..., p n-1 d for each of these n-2 points Δi That is, we get:

[0113] d Δ2 d Δ3 ... d Δn-1 ;

[0114] Step 503, determine the mean value d of all intermediate pixels. Δi mean d s ;

[0115] Step 504, determine the mean d s Is it greater than the preset threshold t? s If yes, proceed to step 505; otherwise, proceed to step 506.

[0116] Step 505: Identify pixels where depth values ​​have changed abruptly.

[0117] Step 506: Determine that there are no pixels with abrupt changes in depth values.

[0118] For example, such as Figure 7 As shown, it mainly includes:

[0119] Step 701, calculate d Δ2 d Δ3 ... d Δn-1 mean d s ;

[0120] Step 702, determine the mean d s Is it greater than the preset threshold t? s If yes, proceed to step 703; otherwise, proceed to step 704.

[0121] Step 703, if d s ≤t s This indicates that the identification of the key points of the elbow (knee) bones is correct;

[0122] Step 704, if d s >t s If the identification of the key points of the elbow (knee) bones is incorrect, it indicates that the identification of the key points of the elbow (knee) bones is incorrect.

[0123] If d s ≤t s If the identification of the elbow (knee) bone point is correct, then the shoulder (hip) bone point a is replaced with the elbow (knee) bone point a, and the original elbow (knee) bone point b is replaced with the wrist (ankle) bone point. Then, check whether the wrist (ankle) bone point is incorrectly identified.

[0124] like Figure 8 As shown, the skeletal point b at the elbow (knee) T Incorrectly identified as b F There is a depth value mutation site p j ,like Figure 9 As shown, the bony point b of the wrist (ankle) T Incorrectly identified as b F There is a depth value mutation site pj Using the method described in the embodiments of this application, erroneous bone points in the elbow, knee, wrist, and ankle are identified using depth value information. The embodiments of this application identify depth value abrupt change locations p as described above. j The depth value of the mutation site p j Starting from the depth map information, the local boundary is drawn, and then the boundary of the RGB image is drawn accordingly. The bone points are then regenerated using the existing bone point detection algorithm.

[0125] In one or more possible embodiments, for any set of skeletal keypoints, if there is a pixel that experiences a sudden change in depth value, such as Figure 10 As shown, it also includes:

[0126] Step 1001: Determine the depth value change location corresponding to the pixel that experienced the depth value change.

[0127] Step 1002: Taking the depth value abrupt change location as the center, calculate the depth value difference between the depth value abrupt change location and the surrounding neighboring pixels.

[0128] Step 1003: Determine the maximum value of the absolute value of the depth difference as the depth difference threshold.

[0129] like Figure 11 As shown, the depth value abruptly changes at location p. j Centered on the black square, calculate the absolute value of the depth difference between it and its eight neighboring pixels (the squares indicated by the arrows). A total of eight absolute values ​​are calculated. The largest of these eight absolute values ​​(the square with the deepened boundary) is taken as the depth difference threshold, denoted as t. m .

[0130] In one or more possible embodiments, boundary pixels belonging to the human body contour are determined based on the traversal results, such as... Figure 12 As shown, it includes:

[0131] Step 1201: Determine the absolute value of the depth difference between the traversed pixel and any pixel in its surrounding neighborhood. The surrounding neighborhood can be either an 8-neighborhood or a 4-neighborhood.

[0132] Step 1202: Determine if there is a value exceeding the depth difference threshold t. m If the absolute value of the depth difference is true, proceed to step 1203; otherwise, proceed to step 1204.

[0133] Step 1203: Determine which of the traversed pixels belong to the boundary pixels of the human body contour;

[0134] Step 1204: Determine which of the traversed pixels are not boundary pixels of the human body contour.

[0135] In this embodiment of the application, the depth value change location corresponding to the pixel that experiences a depth value change is determined, such as... Figure 13 As shown, it includes:

[0136] Step 1301: Obtain the intermediate pixels p2, p3, ..., p n-1 The corresponding mean d Δ2 d Δ3 ... d Δn-1 ;

[0137] Step 1302: Determine the mean value d of all intermediate pixels. Δ2 d Δ3 ... d Δn-1 The middle pixel corresponding to the maximum value is the location of the depth value change corresponding to the pixel where the depth value changes abruptly.

[0138] In one or more possible embodiments, a local region is defined based on the adjacent skeletal key points, such as... Figure 14 As shown, it includes:

[0139] Step 1401: Determine the distance L between the adjacent skeletal key points;

[0140] Step 1402: Using the inaccurately identified bone key points among the adjacent bone key points as the center, draw a circle with a radius of w times the distance L, and determine the area composed of all pixels within the circle as the local area.

[0141] For example, using the incorrectly identified elbow (knee) bone point b F Centered on a circle, with the shoulder (hip) bone point a and the incorrectly identified elbow (knee) bone point b... F The circle is centered at a distance w (w≥1), and the region consisting of all pixels within the circle is the local region.

[0142] In one or more possible embodiments, corresponding boundary pixels are marked in the RGB image, such as... Figure 15 As shown, it includes:

[0143] Step 1501: Determine the corresponding boundary pixels in the RGB image;

[0144] Step 1502: Set the color value of the boundary pixels in the RGB image to the complementary color to obtain the labeled RGB image.

[0145] This embodiment traverses all points in the local region according to the above steps, obtaining all boundary points in the local region. In the RGB image, for the pixels corresponding to all boundary points, their RGB color values ​​are set to their complementary colors to enhance the contrast of the boundaries, such as... Figure 16 The black square pixels are shown in the image.

[0146] For example, in this embodiment, the RGB image with the boundary point pixel values ​​set to complementary colors is re-input into the human skeletal keypoint detection algorithm to output the corrected elbow (knee) bone point positions. The wrist (ankle) bone point positions are then regenerated using the same method.

[0147] Based on the same inventive concept, this application also provides a skeletal key point correction device 1700, such as... Figure 17 As shown, the device includes:

[0148] Image acquisition module 1701 is used to acquire RGB images and depth images of the human body, and to perform human body recognition on the RGB images to obtain the skeletal key points of the human body.

[0149] The anomaly determination module 1702 is used to determine whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal key points in the depth image.

[0150] The depth value traversal module 1703 is used to, if there are pixels with a sudden change in depth value, delineate a local region based on the adjacent skeletal key points, traverse the pixels in the local region, determine the traversed pixels and the adjacent pixels adjacent to the traversed pixels, and determine the depth value difference between the traversed pixels and the adjacent pixels.

[0151] The boundary determination module 1704 is used to determine the boundary pixels belonging to the human body contour based on the traversal results, and mark the corresponding boundary pixels in the RGB image.

[0152] The key point correction module 1705 is used to re-identify the human body in the annotated RGB image to obtain the corrected skeletal key points of the human body.

[0153] In one or more possible embodiments, the anomaly determination module determines whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal keypoints in the depth image, including:

[0154] By connecting adjacent skeletal key points in the depth image as endpoints, multiple pixels located on the connecting lines in the depth image are determined.

[0155] For any intermediate pixel p among the plurality of pixels, excluding the first and last pixels... i Calculate the mean d of the absolute values ​​of the depth differences between the middle pixel and its two adjacent pixels. Δi ;

[0156] Determine the mean d of all intermediate pixels. Δi mean d s If the mean ds Greater than the preset threshold t s At that time, it was determined that there were pixels where the depth value changed abruptly.

[0157] In one or more possible embodiments, such as Figure 18 As shown, if there are pixels where the depth value changes abruptly, the device further includes:

[0158] The threshold determination module 1706 is used to determine the depth value change location corresponding to the pixel point where the depth value change occurs; with the depth value change location as the center, the depth value difference between the depth value change location and the surrounding neighboring pixels is calculated, and the maximum value of the absolute value of the depth value difference is determined as the depth value difference threshold.

[0159] The boundary determination module determines the boundary pixels belonging to the human body contour based on the traversal results, including:

[0160] When the absolute value of the depth difference between a traversed pixel and any neighboring pixel exceeds the depth difference threshold, the traversed pixel is determined to be a boundary pixel of the human body contour.

[0161] In one or more possible embodiments, the threshold determination module determines the depth value change location corresponding to the pixel where the depth value change occurs, including:

[0162] Determine the mean d of all intermediate pixels. Δi The middle pixel corresponding to the maximum value is the location of the depth value change corresponding to the pixel where the depth value changes abruptly.

[0163] In one or more possible embodiments, the depth traversal module delineates a local region based on the adjacent skeletal keypoints, including:

[0164] Determine the distance L between the adjacent skeletal key points;

[0165] Using the inaccurately identified bone key points among the adjacent bone key points as the center, draw a circle with a radius of w times the distance L, and determine the area composed of all pixels within the circle as the local region.

[0166] In one or more possible embodiments, the boundary determination module marks the corresponding boundary pixels in the RGB image, including:

[0167] Determine the corresponding boundary pixels in the RGB image;

[0168] The color values ​​of the boundary pixels in the RGB image are set to their complementary colors to obtain the annotated RGB image.

[0169] In one or more possible embodiments, the adjacent skeletal key points include a first group of skeletal key points consisting of shoulder and elbow skeletal key points, a second group of skeletal key points consisting of elbow and wrist skeletal key points, a third group of skeletal key points consisting of hip and knee skeletal key points, and a fourth group of skeletal key points consisting of knee and ankle skeletal key points, wherein:

[0170] The first and third sets of skeletal key points are identified sequentially, and then the second and fourth sets of skeletal key points are identified.

[0171] If there are no pixels with abrupt changes in depth value in the first and third groups of skeletal keypoints during the current traversal, traverse the second and fourth groups of skeletal keypoints. Otherwise, annotate the RGB image once, perform human recognition on the annotated RGB image to correct the first or third group of skeletal keypoints with recognition errors, and then traverse the second and fourth groups of skeletal keypoints after correction.

[0172] If there are pixels with abrupt changes in depth value in the second and fourth sets of skeletal keypoints during the current traversal, annotate the RGB image once, and perform human recognition on the annotated RGB image to correct the second or fourth set of skeletal keypoints with recognition errors.

[0173] If there are pixels with a sudden change in depth value in the first group of skeletal key points / the third group of skeletal key points being traversed, then the identification of elbow / knee skeletal key points is determined to be inaccurate.

[0174] If there are pixels with abrupt changes in depth value in the second or fourth set of skeletal keypoints during the current traversal, then the identification of wrist or ankle skeletal keypoints is determined to be inaccurate.

[0175] In one or more possible embodiments, calculating the depth value difference between the depth value abrupt change region and its surrounding neighboring pixels, centered on the depth value abrupt change region, includes:

[0176] Centered on the location of the abrupt change in depth value, calculate the depth value difference between the location of the abrupt change and the four surrounding neighboring pixels; or

[0177] Centered on the depth value abrupt change location, calculate the depth value difference between the depth value abrupt change location and the surrounding eight neighboring pixels.

[0178] Having described the skeletal keypoint correction method and apparatus according to exemplary embodiments of this application, we will now describe an electronic device according to another exemplary embodiment of this application.

[0179] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0180] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of the skeletal keypoint correction method according to the various exemplary embodiments of this application described above. For example, the processor may perform steps such as those in the skeletal keypoint correction method.

[0181] The following reference Figure 19 To describe an electronic device 190 according to this embodiment of the present application. Figure 19 The electronic device 190 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 19 As shown, the electronic device 190 is presented in the form of a general-purpose electronic device. The components of the electronic device 190 may include, but are not limited to: at least one processor 191, at least one memory 192, and a bus 193 connecting different system components (including memory 192 and processor 191).

[0183] Bus 193 represents one or more of several bus architectures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus architectures.

[0184] The memory 192 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1921 and / or cache memory 1922, and may further include read-only memory (ROM) 1923.

[0185] The memory 192 may also include a program / utility 1925 having a set (at least one) of program modules 1924, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0186] Electronic device 190 can also communicate with one or more external devices 194 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 190, and / or with any device that enables electronic device 190 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 195. Furthermore, electronic device 190 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 196. As shown, network adapter 196 communicates with other modules used in electronic device 190 via bus 193. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 190, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0187] In some possible implementations, various aspects of the skeletal keypoint correction method provided in this application can also be implemented in the form of a program product, which includes program code that, when the program product is run on a computer device, causes the computer device to perform the steps of the skeletal keypoint correction method according to the various exemplary embodiments of this application described above.

[0188] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0189] The program product for skeletal keypoint correction according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0190] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0191] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0192] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0193] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0194] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0195] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] This application is described with reference to flowchart illustrations and block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block and / or block in the flowchart illustrations and block diagrams, as well as combinations of blocks and processes in the flowchart illustrations and block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0197] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and boxes Figure 1 The function specified in one or more boxes.

[0198] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and boxes Figure 1 The steps of the function specified in one or more boxes.

[0199] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0200] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for correcting skeletal key points, characterized in that, The method includes: Acquire RGB and depth images of the human body, and perform human body recognition on the RGB images to obtain the key skeletal points of the human body; Based on the depth values ​​of pixels between adjacent skeletal key points in the depth image, determine whether there are pixels with abrupt changes in depth values; If there are pixels with a sudden change in depth value, a local region is defined based on the adjacent skeletal key points. The pixels in the local region are traversed to determine the traversed pixels and the adjacent pixels adjacent to the traversed pixels, and the depth value difference between the traversed pixels and the adjacent pixels is determined. Based on the traversal results, the boundary pixels belonging to the human body contour are determined, and the corresponding boundary pixels are marked in the RGB image. The annotated RGB image is re-performed for human body recognition to obtain the corrected skeletal key points of the human body; The step of determining whether there are pixels with abrupt changes in depth value based on the depth values ​​of pixels between adjacent skeletal keypoints in the depth image includes: By connecting adjacent skeletal key points in the depth image as endpoints, multiple pixels located on the connecting lines in the depth image are determined. For any intermediate pixel p among the plurality of pixels, excluding the first and last pixels... i Calculate the mean d of the absolute values ​​of the depth differences between the middle pixel and its two adjacent pixels. Δi ; Determine the mean d of all intermediate pixels. Δi mean d s If the mean d s Greater than the preset threshold t s At that time, it was determined that there were pixels where the depth value changed abruptly; The process of defining a local region based on the adjacent skeletal key points includes: Determine the distance L between the adjacent skeletal key points; Using the inaccurately identified bone key points among the adjacent bone key points as the center, draw a circle with a radius of w times the distance L, and determine the area composed of all pixels within the circle as the local region.

2. The method according to claim 1, characterized in that, If there are pixels where the depth value changes abruptly, it also includes: Identify the location of the depth value abrupt change corresponding to the pixel where the depth value abruptly changes; Centered on the depth value abrupt change location, calculate the depth value difference between the depth value abrupt change location and the surrounding neighboring pixels, and determine the maximum absolute value of the depth value difference as the depth value difference threshold. The step of determining the boundary pixels belonging to the human body contour based on the traversal results includes: When the absolute value of the depth difference between a traversed pixel and any neighboring pixel exceeds the depth difference threshold, the traversed pixel is determined to be a boundary pixel of the human body contour.

3. The method according to claim 2, characterized in that, Identify the depth value abrupt change location corresponding to the pixel where the depth value abrupt change occurs, including: Determine the mean d of all intermediate pixels. Δi The middle pixel corresponding to the maximum value is the location of the depth value change corresponding to the pixel where the depth value changes abruptly.

4. The method according to claim 1, characterized in that, Marking the corresponding boundary pixels in the RGB image includes: Determine the corresponding boundary pixels in the RGB image; The color values ​​of the boundary pixels in the RGB image are set to their complementary colors to obtain the annotated RGB image.

5. The method according to claim 1, characterized in that, The adjacent skeletal key points include a first group of skeletal key points consisting of shoulder and elbow skeletal key points, a second group of skeletal key points consisting of elbow and wrist skeletal key points, a third group of skeletal key points consisting of hip and knee skeletal key points, and a fourth group of skeletal key points consisting of knee and ankle skeletal key points, wherein: The first and third sets of skeletal key points are identified sequentially, and then the second and fourth sets of skeletal key points are identified. If there are no pixels with abrupt changes in depth value in the first and third groups of skeletal keypoints during the current traversal, traverse the second and fourth groups of skeletal keypoints. Otherwise, annotate the RGB image once, perform human recognition on the annotated RGB image to correct the first or third group of skeletal keypoints with recognition errors, and then traverse the second and fourth groups of skeletal keypoints after correction. If there are pixels with abrupt changes in depth value in the second and fourth sets of skeletal keypoints during the current traversal, annotate the RGB image once, and perform human recognition on the annotated RGB image to correct the second or fourth set of skeletal keypoints with recognition errors.

6. The method according to claim 5, characterized in that, If there are pixels with a sudden change in depth value in the first group of skeletal key points / the third group of skeletal key points being traversed, then the identification of elbow / knee skeletal key points is determined to be inaccurate. If there are pixels with abrupt changes in depth value in the second or fourth set of skeletal keypoints during the current traversal, then the identification of wrist or ankle skeletal keypoints is determined to be inaccurate.

7. The method according to claim 2, characterized in that, Centered on the location of the abrupt change in depth value, the depth value difference between the location of the abrupt change and the surrounding neighboring pixels is calculated, including: Centered on the location of the abrupt change in depth value, calculate the depth value difference between the location of the abrupt change and the four neighboring pixels; or Centered on the depth value abrupt change location, calculate the depth value difference between the depth value abrupt change location and the surrounding eight neighboring pixels.

8. An electronic device, characterized in that, The device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the skeletal keypoint correction method as described in any one of claims 1-7.

Images