A monocular color video-based 3D human joint position locating method

Abstract

The present application relates to the technical field of computer vision, in particular to a three-dimensional human body joint positioning method of monocular color video, the present application obtains monocular color video images and pre-processes the obtained monocular color video images; extracts human target monocular color video images from the pre-processed monocular color video images, and carries out binaryzation processing on the human target monocular color video images to obtain binaryzation depth human target monocular color video images; inputs the obtained binaryzation depth human target monocular color video images into a skeleton feature enhancement graph convolution network to extract human skeleton data, and obtains static features and time dynamic features based on the extracted human skeleton data; fuses the static features and the time dynamic features, extracts high-level motion features based on the fusion result, and generates behavior recognition results, which helps to improve the recognition accuracy of the behavior recognition results.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional human body joint positioning method and a storage medium for a monocular color video. Background Art

[0002] Human posture estimation has always been an important research direction in the field of computer vision and the foundation of many other research fields. For example, behavior analysis, gait recognition, and person tracking all require the precise coordinate positions of joint points for further analysis and judgment.

[0003] Currently, existing neural network models often misclassify certain behaviors when classifying motion behaviors, causing correct classifications to be misclassified as other behaviors. Furthermore, the recognition accuracy is too low. For example, the key features of right waving (Wav), right-hand knocking (Kno), and right-hand reaching out to grasp an object (RCa) are concentrated in the arms, which are easily obscured by the body, making it difficult to distinguish motion movements. Summary of the Invention

[0004] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a three-dimensional human joint point positioning method for monocular color video, which helps to extract the human skeleton diagram when the arms are occluded during movement, thereby improving the accuracy of recognition.

[0005] In order to implement the above technical solution, in a first aspect, the present invention provides a method for locating three-dimensional human joints based on a monocular color video, comprising: step 1: acquiring a monocular color video image and preprocessing the acquired monocular color video image;

[0006] Step 2: extracting a human target monocular color video image from the preprocessed monocular color video image, and performing binarization processing on the human target monocular color video image to obtain a binary depth human target monocular color video image;

[0007] Step 3: Extract the human skeleton including joint points from the binary depth human target monocular color video image;

[0008] Step 4: Input the obtained human skeleton image into the motion recognition model to determine the motion recognition result;

[0009] The step three includes: using a diamond-shaped four-neighborhood template to perform the same number of corrosion and expansion operations on the binary depth human target monocular color video image to achieve human body background segmentation, thereby obtaining the human body area range;

[0010] Counting the width of the obtained human body region, and scanning the determined human body region to determine the human body shoulder joint coordinate information and hip key coordinate information;

[0011] Based on the coordinate information of the hip joint, skeleton extraction is performed on the binary depth human target monocular color video image to extract the overall human skeleton map;

[0012] Extract the arm skeleton based on the coordinate information of the two shoulder joints;

[0013] Combining the obtained arm skeleton diagram with the whole human skeleton diagram to obtain a human skeleton diagram;

[0014] Based on the determined human shoulder joint coordinate information and hip key coordinate information and the human skeleton diagram, determining coordinate information of other human joints;

[0015] The step 4 includes: extracting human skeleton data from the binary depth human target monocular color video image through a human posture estimation algorithm, and obtaining static features and temporal dynamic features based on the extracted human skeleton data;

[0016] Fuse static features and temporal dynamic features, and extract high-level motion features based on the fusion results;

[0017] Generate action recognition results based on high-level motion features.

[0018] Furthermore, the step 2 includes:

[0019] Perform filtering on the acquired monocular color video image;

[0020] Perform image enhancement on the filtered monocular color video image.

[0021] Furthermore, the step 2 includes:

[0022] Customize a rectangular frame containing a human target in the monocular color video image after image enhancement processing;

[0023] A Gaussian mixture model is used to model the background within a custom rectangular frame to obtain a monocular color video image of the human target;

[0024] The segmented human target monocular color video image is binarized to obtain a binary depth human target monocular color video image.

[0025] Furthermore, the filtering process of the acquired monocular color video image includes:

[0026] A: Construct a filter window Sxy of size S;

[0027] B: Count the grayscale values ​​of all pixels in the filter window and determine the grayscale median, maximum and minimum values ​​of all pixels;

[0028] C: Calculate the difference A1 between the grayscale median and the grayscale maximum value, and the difference A2 between the grayscale median and the grayscale minimum value;

[0029] D: If the difference A1 is not greater than zero and / or the difference A2 is not less than zero, then if S is less than or equal to Smax, construct a filter window of size S plus 2 and repeat step B;

[0030] E: If the difference A1 is greater than zero and the difference A2 is less than zero, then perform the following steps:

[0031] (1) Determine the grayscale value of the pixel at the position (x, y) within the filter window Sxy with a window size of S;

[0032] (2) Calculate the difference B1 between the grayscale value of the pixel at the (x, y) position and the maximum grayscale value, and the difference B2 between the grayscale value of the pixel at the (x, y) position and the minimum grayscale value;

[0033] If the difference B1 is not greater than zero and / or B2 is not less than zero, the median pixel grayscale value is used as the pixel grayscale value at the (x, y) position.

[0034] Furthermore, the image enhancement processing of the filtered monocular color video image includes:

[0035] Divide the filtered monocular color video image into several sub-blocks;

[0036] Each sub-block is cropped according to the preset histogram contrast threshold, and the cropped part is redistributed evenly throughout the grayscale range.

[0037] In a second aspect, the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for locating three-dimensional human joints in a monocular color video.

[0038] The beneficial effects of the present invention are:

[0039] (1) The present invention uses a diamond-shaped four-neighborhood template to perform the same number of corrosion and dilation operations on a binary depth human target monocular color video image, thereby obtaining the human body area range; and scans the determined human body area range to determine the human shoulder joint coordinate information and hip key coordinate information; extracts the human body overall skeleton diagram and the arm part skeleton based on the coordinate information of the hip joint and the coordinate information of the two shoulder joints, and then combines them to obtain a human body skeleton diagram, which helps to avoid the inability to recognize arm movements due to human body occlusion, and locates the joint points of the human body skeleton diagram, thereby obtaining a human body joint point skeleton diagram, which provides reliable data for the recognition of motion state and helps to improve the accuracy of recognition.

[0040] (2) The present invention obtains static features (joint positions, bone vectors, and bone cosine angles) and temporal dynamic features (joint velocities, bone velocities, and skeleton graph velocities), and then adopts a dual-stream branch input structure to perform feature fusion in the spatiotemporal graph convolution block. The fused features are then integrated into a three-layer network for training to extract high-level motion features. Based on the high-level motion features, the behavior recognition results are generated, which not only reduces the number of model parameters and computational complexity, but also further helps to improve the accuracy of the behavior recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0042] Figure 1 This is a flow chart of a method for locating three-dimensional human joints using monocular color video according to the present invention;

[0043] Figure 2 The figure is a flowchart of the human body background segmentation of the present invention, wherein (a) is a binary depth human target monocular color video image; (b) is an image after the binary depth human target monocular color video image is eroded; (c) is an image after the binary depth human target monocular color video image is expanded;

[0044] Figure 3 A human skeleton diagram of the present invention;

[0045] Figure 4 This is a flowchart of the arm skeleton extraction process of the present invention, wherein (a) is the preliminary arm silhouette of the present invention; (b) is the arm silhouette of the present invention; (c) is the arm skeleton diagram of the present invention;

[0046] Figure 5 A combined skeleton diagram of the body and arm skeletons of the present invention;

[0047] Figure 6This is a flow chart of the human skeleton positioning process of the present invention, wherein (a) is a rendering of the spatial coordinate information of the human hand and elbow; (b) is a rendering of the spatial coordinate information of the human skeleton; (c) is a human skeleton diagram of the joint points;

[0048] Figure 7 Schematic diagram of the motion recognition model structure of the present invention;

[0049] Figure 8 Schematic diagram of the dual-stream branch input structure of the motion recognition model of the present invention;

[0050] Figure 9 A comparison of the training loss value versus Epoch curves of the motion recognition model of the present invention and the existing model;

[0051] Figure 10 The figure is a comparison of the test accuracy curves of the motion recognition model of the present invention and the existing model with Epoch change. DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, each technical and scientific term used in this embodiment has the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0054] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0055] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention, and should not be understood as limiting the present invention.

[0056] In the present invention, terms such as "fixed connection," "connected," and "connection" should be interpreted broadly to mean a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediary. Relevant researchers or technicians in this field may determine the specific meanings of these terms in the present invention based on specific circumstances, and they should not be construed as limitations of the present invention.

[0057] Example 1:

[0058] like Figure 1 As shown, this embodiment provides a method for locating three-dimensional human joints in a monocular color video, comprising the following steps:

[0059] S1: Acquire a monocular color video image.

[0060] S2: Preprocess the acquired monocular color video image.

[0061] The specific steps include:

[0062] S2-1: Filter the acquired monocular color video image.

[0063] The specific steps include:

[0064] A: Construct a filter window Sxy of size S;

[0065] B: Count the grayscale values ​​of all pixels in the filter window and determine the grayscale median, maximum and minimum values ​​of all pixels;

[0066] C: Calculate the difference A1 between the grayscale median and the grayscale maximum value, and the difference A2 between the grayscale median and the grayscale minimum value; the calculation formula is as follows:

[0067] A1=Zmed-Zmin;

[0068] A2=Zmed-Zmax;

[0069] D: If the difference A1 is not greater than zero and / or the difference A2 is not less than zero, then if S is less than or equal to Smax, construct a filter window of size S plus 2 and repeat step B;

[0070] E: If the difference A1 is greater than zero and the difference A2 is less than zero, then perform the following steps:

[0071] (1) Determine the grayscale value of the pixel at the position (x, y) within the filter window Sxy with a window size of S;

[0072] (2) Calculate the difference B1 between the grayscale value of the pixel at the (x, y) position and the maximum grayscale value, and the difference B2 between the grayscale value of the pixel at the (x, y) position and the minimum grayscale value;

[0073] The specific calculation formula is as follows:

[0074] B1=Zxy-Zmin;

[0075] B2=Zxy-Zmax;

[0076] If the difference B1 is not greater than zero and / or B2 is not less than zero, the median pixel grayscale value is used as the pixel grayscale value at the (x, y) position.

[0077] Through the adaptive median filtering process, the size of the filtering window can be adaptively changed according to the noise density. A large window is used in areas with a higher frequency of noise to improve the purity of the picture, and a small window is used in areas with a lower frequency of noise to help preserve the details of the monocular color video image.

[0078] S2-2: Perform image enhancement processing on the filtered monocular color video image.

[0079] The specific steps include:

[0080] A: Divide the filtered monocular color video image into several sub-blocks.

[0081] B: Perform histogram equalization on each sub-block separately.

[0082] The specific steps include:

[0083] (1) Preset the contrast threshold of the histogram;

[0084] (2) Each sub-block is cropped according to the preset histogram contrast threshold, and the cropped part is redistributed evenly in the entire grayscale range.

[0085] For example, assuming the contrast threshold (cropping value) is T, the sum of the grayscale portions of each sub-block greater than T is recorded as Sum, and then Sum is evenly distributed throughout the entire grayscale range, so that the filtered monocular color video image as a whole rises by a height L = Sum / N, where N represents the total number of pixels in the monocular color video image. The histogram is processed as follows with H = TL as the boundary:

[0086] 1) If the histogram amplitude is greater than T, its value is directly set to T;

[0087] 2) If the histogram amplitude is between H and T, fill it to T;

[0088] 3) If the histogram amplitude is less than H, then fill L pixels directly.

[0089] S3: extracting a human target monocular color video image from the preprocessed monocular color video image, and performing binarization processing on the human target monocular color video image.

[0090] The specific steps include:

[0091] S3-1: Customize a rectangular box containing a human target.

[0092] S3-2: Use the Gaussian mixture model (GMM) to model the background within the custom rectangular frame to obtain the background and foreground segmentation results, that is, to obtain a monocular color video image of the human target.

[0093] Specifically, in a monocular color video image, pixels outside the rectangular frame are assumed to be background. Each pixel in the monocular color video image is considered to be connected to surrounding pixels by an edge, and each edge is assigned a probability of belonging to the foreground or background based on its color similarity with the surrounding pixels. Finally, if the edge between two pixels belongs to different categories (foreground and background), the edge between them is cut off, thus segmenting the monocular color video image.

[0094] S3-3: Binarize the segmented human target monocular color video image to obtain a binary depth human target monocular color video image.

[0095] S4: Extracting a human skeleton graph including joint points from a binarized depth human target monocular color video image.

[0096] The human body skeleton data is extracted from the binary depth human target monocular color video image using a human body pose estimation algorithm. The human body skeleton data includes the spatial coordinate information of each joint point of the human body in different frames. The specific steps include:

[0097] A-1: Using a diamond-shaped four-neighborhood template, perform the same number of erosion and dilation operations on the binary depth human target monocular color video image to achieve human background segmentation, thereby obtaining the human area range, (such as Figure 2 As shown, Figure 2 (a) in the figure represents a binary depth monocular color video image of a human target. Figure 2 (b) in the figure represents the image after the binary depth human target monocular color video image is corroded. Figure 2 (c) in the figure shows the image after the binary depth human target monocular color video image is expanded).

[0098] The erosion operation removes the thinner legs, head, and unobstructed hand silhouettes, while retaining the larger central portion of the body. Following the erosion, the same dilation operation is performed to restore the original body, while the rest of the body is completely eliminated. This effectively removes the effects of internal noise and uneven boundaries, resulting in an extracted skeleton with fewer unrealistic branches.

[0099] It should be noted that the corrosion operation and dilation operation implement corrosion as follows:

[0100] Q1: Assume that X and Y are two sets in a two-dimensional Euclidean space Ώ, and x, y, and z are points in the Euclidean space. The following formula defines the set after the elements in the X set are transformed when point y is used as the origin:

[0101] .

[0102] Q2: Perform expansion operation using the following formula:

[0103] ;

[0104] Here, U represents the result of X being expanded by Y. Dilation is the union of the graph X after translating it by each value of y. If Y is symmetric about the origin, Y is moved at the boundary of X, and the covered area is also incorporated into U. The intuitive effect is that X becomes larger, so it is called a dilation operation.

[0105] Q3: Perform corrosion operation using the following formula:

[0106] ;

[0107] Here, Z represents the result of X being eroded by Y. The effect of the erosion operation is that the elements in Z are translated by the template Y to form A. If the template Y is symmetric about the origin, then the final effect is to remove some elements outside A to form Z. The intuitive effect is that X becomes smaller.

[0108] A-2: Count the width of the obtained human body region and scan the determined human body region to determine the human body shoulder joint coordinate information and hip key coordinate information.

[0109] The specific steps include:

[0110] (1) The width of the obtained human body region is counted to obtain the average width value W, and the average width value W is used as the width of the human shoulder joint. Based on the proportional relationship between the pixel and the actual image, the pixel value W+n (for example, W+6) is obtained.

[0111] When performing width statistics on the human body area, significant minimum values ​​and significant maximum values ​​are automatically removed.

[0112] (2) The human body area is scanned from top to bottom. When the width of the human body area in several consecutive rows is greater than the average width value W, it is determined as the longitudinal coordinate position of the shoulder joint, that is, the height position of the shoulder joint is determined.

[0113] (3) Based on the highest point angle, determine the horizontal coordinate position of the midpoint of the line connecting the two shoulders, that is, determine the horizontal position of the shoulder joint.

[0114] (4) Based on the determined longitudinal coordinate position of the shoulder joint and the transverse coordinate position of the center, the coordinate information of the two shoulder joints in the human body region is determined.

[0115] (5) In a similar way, the human body area is scanned from top to bottom. When it is detected that the width of several consecutive rows of human body areas is close to the average width value W, the longitudinal coordinate position of the hip joint is determined, that is, the height position of the hip joint is determined. The boundary of the human body area at the height position is the coordinate information of the hip joint.

[0116] A-3: Based on the coordinate information of the hip joint, skeleton extraction is performed on the binary depth human target monocular color video image to extract the overall human skeleton map ( Figure 3 shown).

[0117] A-4: Extract the arm skeleton based on the coordinate information of the two shoulder joints.

[0118] like Figure 4 As shown, the specific steps include:

[0119] (1) The range of human body area (i.e. Figure 2 The depth values ​​within the human body range shown in (c) are counted, and the depth values ​​that differ by more than 3 from the average value are removed, that is, the influence of the arm blocking the front is removed, and the remaining depth values ​​are averaged, and the average depth value is recorded as AVE_body.

[0120] (2) Set the pixels whose depth values ​​within the human body area differ from AVE_body by less than 4 to zero, and retain the other values ​​to obtain the preliminary arm silhouette (e.g. Figure 4 (a) in the figure.

[0121] (3) Perform several erosion operations on the arm silhouette to remove small interference areas and obtain the arm silhouette (such as Figure 4 (b) in the figure).

[0122] (4) Extract the skeleton of the obtained arm silhouette to obtain the arm skeleton diagram (such as Figure 4 (c) in the figure).

[0123] A-5: Combine the obtained arm skeleton image and the human body skeleton image to obtain the human body skeleton image (such as Figure 5 ).

[0124] A-6: Based on the determined human shoulder joint coordinate information and hip key coordinate information and the human skeleton diagram, determine the coordinate information of other human joints.

[0125] Among them, other joints include: head, neck, hips, feet, legs, elbows, hands and knees.

[0126] The specific steps include:

[0127] (1) Based on the coordinate information of the two shoulder joints and the coordinate information of the hip joints and the human skeleton model, the coordinate information of the head, neck, hips, feet, and legs can be directly determined.

[0128] Specifically, since the neck is the midpoint of the line connecting the two shoulder joints, the head is the endpoint of the skeleton above the neck, the hips are the bifurcation points connecting the two legs and the body, and the two feet are the two lowest endpoints of the skeleton; and since the position of the hip joint is the width of the body, when the hip joint information is known, the hips are located on the center line of the hip joint.

[0129] (2) Based on the coordinate information of the two shoulder joints and the coordinate information of the hip joint, the coordinate information of the elbow, hand and knee is determined using the maximum triangle method.

[0130] The specific steps include:

[0131] K1: Start searching from the shoulder joint and find a point closer to the shoulder joint as the point on the upper arm, and the farther point as the hand.

[0132] K2: Calculate the coordinates of all points on the skeleton line in real space, and find a point on the skeleton line that maximizes the area of ​​the triangle formed by this point, the hand, and the shoulder in real 3D space. This point is used as the turning point on the hand skeleton line, i.e., the coordinates of the elbow joint. If the areas are very small, then the arm is basically straight, and the midpoint of the skeleton line is used as the coordinates of the elbow joint (e.g. Figure 6 (as shown in (a) in the figure).

[0133] K3: Use the above steps to determine the coordinate information of the knee joint point of the leg.

[0134] (3) Based on the determined coordinate information of the head, neck, hips, feet, legs, elbows, hands and knees, obtain the spatial coordinate information of the human skeleton (such as Figure 6 (as shown in (b) in the figure).

[0135] (4) Connect the acquired human skeleton spatial coordinate information with straight lines to form a human skeleton diagram including joint points (such as Figure 6 (as shown in (c) in the figure).

[0136] Step S4 enables the correct segmentation of the human body region in a complex background environment, helps eliminate the influence of internal noise and uneven boundary areas in the human body region, and avoids the phenomenon of being unable to obtain the correct joint point human skeleton due to body occlusion.

[0137] S5: Input the obtained human skeleton image including the joint points into the motion recognition model to determine the motion recognition result.

[0138] Among them, Figure 7 As shown, the motion recognition model includes: four layers of networks, which are stacked; the four layers of networks are respectively a first layer of network, a second layer of network, a third layer of network and a fourth layer of network.

[0139] The first-layer network is a spatiotemporal graph convolution module, which is used to extract the static and temporal dynamic features of skeleton behavior data from the human skeleton graph. The basic network structure mainly includes spatial graph convolution blocks and temporal convolution blocks, which respectively extract the static and dynamic characteristics of skeleton behavior. Among them, the dynamic features include: the velocity features of joint points, the velocity features of bone edges, and the body velocity features of the entire human skeleton.

[0140] The second, third and fourth layers of the network are the global adaptation and local enhancement graph convolutional network module (GL-GCN).

[0141] In addition, the static features and dynamic features extracted by the output of the first layer of the network are fused before being input into the second layer of the network. The fused results are then sent to the second, third and fourth layers of the network to extract high-level motion features.

[0142] The following steps are as follows:

[0143] S5-1: Extract static and dynamic features of skeleton behavior data from human skeleton graph.

[0144] The specific steps include:

[0145] E1: Extracting static features of skeleton behavior data from the human skeleton graph specifically includes the following steps:

[0146] (1) Assume that the spatial coordinate information of the joint point is represented as S:

[0147] ;

[0148] Among them, C represents the number of channels; T represents the number of frames of the human skeleton sequence; and V represents the number of joint points in the human skeleton graph.

[0149] (2) The joint point coordinate information is normalized through the normalization module, which specifically includes the following steps:

[0150] Determine P as the average Euclidean distance from one joint point to another joint point in the human skeleton from the nth frame to the mth frame. The formula is as follows:

[0151] .

[0152] (3) Normalized joint point spatial coordinate information set:

[0153] .

[0154] E2: Extracting dynamic features of skeleton behavior data from the human skeleton graph specifically includes the following steps:

[0155] (1) The difference between the adjacent frame and the interval frame is used to calculate the velocity characteristics of the joint points and the velocity characteristics of the bones.

[0156] Among them, the velocity changes of joints and bones can only reflect the local motion trend of the skeleton behavior.

[0157] (2) The global skeleton graph velocity is represented by the difference between the bone vectors formed from the shoulder center joint point to the hip center joint point in the previous frame skeleton graph and the next frame skeleton graph.

[0158] S5-2: Dynamic features and static features are preliminarily fused through a dual-split branch structure, and the fusion results are sent to the second layer network, the third layer network and the fourth layer network in turn.

[0159] Among them, the dual-stream branch input structure is as follows Figure 8 shown.

[0160] Among them, the number of input and output channels and the step size of the first layer network are 6, 16, and 1 respectively.

[0161] The present invention adopts a dual-stream branch input structure for early feature-level fusion, which greatly reduces the calculation of the parameter amount.

[0162] Global average pooling module: used to unify the motion features output by the fourth layer network.

[0163] The fully connected layer and the softmax function layer constitute a prediction network, which is used to perform motion recognition and prediction based on the unified motion features after global average pooling to output motion and prediction results.

[0164] The present invention obtains static features (joint positions, bone vectors, bone cosine angles) and temporal dynamic features (joint velocities, bone velocities, skeleton graph velocities), then adopts a dual-stream branch input structure to perform feature fusion in the spatiotemporal graph convolution block, and then integrates the fused features into a three-layer network for training, thereby reducing the number of model parameters and computational complexity, and improving the recognition accuracy of behavior recognition results.

[0165] Experimental results and analysis:

[0166] The model has a total of four layers of networks. The first layer is a spatiotemporal graph convolution block, and the remaining three layers are global adaptive and local enhanced graph convolution blocks (GL-GCN). The detailed settings of the network layer input and output channels and stride are as follows: Figure 5 All experiments were conducted on a 6-core AMD R5-3500X CPU running at 3.59 GHz, 16 GB of RAM, and an Nvidia GeForce RTX 2060 graphics card with 6 GB of video memory. Model training and testing were performed using the PyTorch 1.5.1 deep learning framework on Windows 10.

[0167] Among them, this experiment compares the correct recognition rate of motion recognition with the Lie group method of traditional manual feature extraction in the existing technology, the multi-view deep motion map (STACOG) method, and the GL-GCN network model composed of three layers of global adaptive and local enhanced graph convolution blocks (GL-GCN). The comparison results are shown in Table 1.

[0168] Table 1 Comparison of recognition accuracy between motion recognition model and existing methods

[0169]

[0170] Overall, the present invention uses a dual-stream branch input structure to fuse rich skeleton behavior features at the feature level in the shallow network, and its recognition accuracy has been improved to a certain extent. From the table above, the GL-GCN network model composed of three layers of global adaptive and local enhanced graph convolution blocks (GL-GCN) has the closest recognition accuracy to the motion recognition model of this embodiment. Taking these two models as examples, the training loss value and test accuracy are compared. Figure 9 The training loss value changes with Epoch as shown in the curve and Figure 10 The test accuracy curve shown here varies with Epoch.

[0171] Depend on Figure 9As shown in the loss value change curves of the motion recognition model of this embodiment and the GL-GCN network model during the same period, the loss value of the motion recognition model of this embodiment changes more rapidly during training, and its loss value drops below 1.0 at the 13th Epoch, while the loss value of the GL-GCN network model does not drop to 0.9015 until the 24th Epoch. This shows that the dual-stream branch input structure that adds early feature fusion of high-order skeleton information has a better fitting effect on skeleton data.

[0172] Depend on Figure 10 As shown in the figure, the motion recognition model of this embodiment has achieved a recognition accuracy of 94.7% in the 45th Epoch, and then achieved the best recognition accuracy of 96.69% in the 85th Epoch. Compared with the GL-GCN network model, whose recognition accuracy fluctuated greatly in the early stage, the motion recognition model of this embodiment is more stable during training.

[0173] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0174] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, and can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0176] In addition, it should be noted that the flowcharts in the accompanying drawings show the methods of the embodiments of the present disclosure. In the descriptions corresponding to the flowcharts or block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be performed substantially in parallel, or sometimes in the opposite order, which may depend on the functions involved. Each block in the block diagram and / or flow chart, and the combination of blocks in the block diagram and / or flow chart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0177] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for locating three-dimensional human joints using monocular color video, characterized in that: include: Step 1: Acquire a monocular color video image and preprocess the acquired monocular color video image; Step 2: extracting a human target monocular color video image from the preprocessed monocular color video image, and performing binarization processing on the human target monocular color video image to obtain a binary depth human target monocular color video image; Step 3: Extract the human skeleton including joint points from the binary depth human target monocular color video image; Step 4: Input the obtained human skeleton image into the motion recognition model to determine the motion recognition result; The step three includes: using a diamond-shaped four-neighborhood template to perform the same number of corrosion and expansion operations on the binary depth human target monocular color video image to achieve human body background segmentation, thereby obtaining the human body area range; Counting the width of the obtained human body region, and scanning the determined human body region to determine the human body shoulder joint coordinate information and hip key coordinate information; Based on the coordinate information of the hip joint, skeleton extraction is performed on the binary depth human target monocular color video image to extract the overall human skeleton map; Extract the arm skeleton based on the coordinate information of the two shoulder joints; Combining the obtained arm skeleton diagram with the whole human skeleton diagram to obtain a human skeleton diagram; Based on the determined human shoulder joint coordinate information and hip key coordinate information and the human skeleton diagram, determining coordinate information of other human joints; The step 4 includes: extracting human skeleton data from the binary depth human target monocular color video image through a human posture estimation algorithm, and obtaining static features and temporal dynamic features based on the extracted human skeleton data; Fuse static features and temporal dynamic features, and extract high-level motion features based on the fusion results; Generate action recognition results based on high-level motion features; The method of counting the width of the obtained human body region and scanning the determined human body region to determine the human body shoulder joint coordinate information and hip key coordinate information includes the following steps: (1) Statistically calculate the width of the human body region to obtain an average width value W, and use the average width value W as the width of the human shoulder joint. Then, based on the ratio of the pixel to the actual image, the pixel value W+n is obtained. When calculating the width of the human body region, the significant minimum value and the significant maximum value are automatically removed. (2) Scan the human body area from top to bottom. When the width of the human body area in several consecutive rows is greater than the average width value W, it is determined as the longitudinal coordinate position of the shoulder joint, that is, the height position of the shoulder joint is determined; (3) Based on the highest point angle, determine the horizontal coordinate position of the midpoint of the line connecting the two shoulders, that is, determine the horizontal position of the shoulder joint; (4) determining the coordinate information of the two shoulder joints in the human body region based on the determined longitudinal coordinate position of the shoulder joint and the transverse coordinate position of the center; (5) Scan the human body area from top to bottom. When it is detected that the width of several consecutive lines of human body areas is close to the average width value W, it is determined as the longitudinal coordinate position of the hip joint, that is, the height position of the hip joint is determined. The boundary of the human body area at the height position is the coordinate information of the hip joint; The method of determining the coordinate information of other joints of the human body based on the determined human shoulder joint coordinate information and hip key coordinate information and the human skeleton diagram, wherein the other joints include: head, neck, hips, two feet, two legs, elbows, hands and knees, comprises the following steps: Based on the determined coordinate information of the two shoulder joints and the coordinate information of the hip joint and the human skeleton model, the coordinate information of the head, neck, hips, feet, and legs is determined; Based on the determined coordinate information of the two shoulder joints and the coordinate information of the hip joint, the coordinate information of the elbow, hand and knee is determined using the maximum triangle method.

2. The method for locating three-dimensional human joints using monocular color video according to claim 1, wherein: The second step includes: Perform filtering on the acquired monocular color video image; Perform image enhancement on the filtered monocular color video image.

3. The method for locating three-dimensional human joints using monocular color video according to claim 1, wherein: The second step includes: Customize a rectangular frame containing a human target in the monocular color video image after image enhancement processing; A Gaussian mixture model is used to model the background within a custom rectangular frame to obtain a monocular color video image of the human target; The segmented human target monocular color video image is binarized to obtain a binary depth human target monocular color video image.

4. The method for locating three-dimensional human joints using monocular color video according to claim 2, wherein: The filtering process of the acquired monocular color video image includes: A: Construct a filter window Sxy of size S; B: Count the grayscale values ​​of all pixels in the filter window and determine the grayscale median, maximum and minimum values ​​of all pixels; C: Calculate the difference A1 between the grayscale median and the grayscale maximum value, and the difference A2 between the grayscale median and the grayscale minimum value; D: If the difference A1 is not greater than zero and / or the difference A2 is not less than zero, then if S is less than or equal to Smax, construct a filter window of size S plus 2 and repeat step B; E: If the difference A1 is greater than zero and the difference A2 is less than zero, then perform the following steps: (1) Determine the grayscale value of the pixel at the position (x, y) within the filter window Sxy with a window size of S; (2) Calculate the difference B1 between the grayscale value of the pixel at the (x, y) position and the maximum grayscale value, and the difference B2 between the grayscale value of the pixel at the (x, y) position and the minimum grayscale value; If the difference B1 is not greater than zero and / or B2 is not less than zero, the median pixel grayscale value is used as the pixel grayscale value at the (x, y) position.

5. The method for locating three-dimensional human joints using monocular color video according to claim 2, wherein: The image enhancement processing of the filtered monocular color video image comprises: Divide the filtered monocular color video image into several sub-blocks; Each sub-block is cropped according to the preset histogram contrast threshold, and the cropped part is redistributed evenly throughout the grayscale range.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the three-dimensional human joint point positioning method for monocular color video according to any one of claims 1 to 5.

Images