Human body tracking method, electronic device, and storage medium
By performing similarity transformation on images and training a key point detection model, the problem of insufficient detection accuracy and speed in existing human body tracking algorithms has been solved, achieving more efficient and accurate human body tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN ZEGO TECH CO LTD
- Filing Date
- 2023-04-27
- Publication Date
- 2026-05-22
AI Technical Summary
Existing human tracking algorithms are insufficient in terms of detection accuracy and speed, leading to inaccurate data analysis.
By acquiring the image to be detected, human detection and feature extraction are performed. The image is transformed using a similarity transformation matrix. Human alignment and position calculation are performed by combining a keypoint detection model. The keypoint detection network is trained using a linear regression method to improve detection accuracy and speed.
It improves the detection accuracy and speed of human body tracking, ensures the accuracy of the target human body position, and reduces the need for repeated detection of each frame of image.
Smart Images
Figure CN116597471B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning, and more particularly to a human body tracking method, electronic device, and storage medium. Background Technology
[0002] Human body tracking algorithms are applied in various scenarios, such as electronic surveillance, personnel density calculation, and intelligent traffic management. However, existing human body tracking algorithms suffer from poor detection accuracy and speed. If human body detection and tracking cannot be performed efficiently and accurately, the data analyzed and calculated in the above scenarios may be inaccurate. Summary of the Invention
[0003] In view of the above, it is necessary to provide a human body tracking method, electronic device and storage medium to improve the detection speed and accuracy of human body tracking.
[0004] On one hand, this application provides a human body tracking method, the method comprising: acquiring an image to be detected; performing human body detection on the image to be detected to obtain multiple detection key points in the image to be detected and detection coordinates of each detection key point; obtaining preset coordinates of preset key points corresponding to each detection key point; transforming the image to be detected according to a similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain a human body aligned image; if a human body is detected in the human body aligned image, inputting the human body aligned image into a preset key point detection model to obtain multiple initial key points in the human body aligned image and initial coordinates of each initial key point; calculating the target coordinates of a target key point corresponding to each initial key point in the image to be detected based on the similarity transformation matrix and the multiple initial coordinates; and calculating the target human body position in the image to be detected based on the preset key points and the multiple target coordinates.
[0005] In some embodiments, the human detection of the human body aligned image includes: extracting features from the human body aligned image to obtain a feature map, predicting human bodies from the feature map to obtain a predicted probability that the human body aligned image contains a human body, and determining that the human body aligned image contains a human body if the predicted probability is greater than or equal to a preset threshold, or determining that the human body aligned image does not contain a human body if the predicted probability is less than the preset threshold.
[0006] In some embodiments, transforming the image to be detected according to a plurality of detection coordinates and a plurality of preset coordinates to obtain a human body aligned image includes: calculating the similarity transformation matrix according to the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates, and translating the image to be detected according to the similarity transformation matrix to obtain the human body aligned image.
[0007] In some embodiments, calculating the similarity transformation matrix based on the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates includes: constructing a vector equation based on the detection vector, the preset vector, the preset parameter matrix corresponding to the detection vector, and the parameter inverse matrix of the preset parameter matrix, wherein the preset parameter matrix includes a plurality of preset parameters, solving the vector equation using the least squares method or the singular value decomposition method to obtain the parameter value corresponding to each preset parameter in the preset parameter matrix, and replacing each preset parameter in the preset parameter matrix with the corresponding parameter value to obtain the similarity transformation matrix.
[0008] In some embodiments, calculating the target coordinates of each initial key point in the image to be detected based on the similarity transformation matrix and the plurality of initial coordinates includes: obtaining the transformation inverse matrix of the similarity transformation matrix, calculating the target coordinates corresponding to each initial coordinate according to each initial coordinate and the transformation inverse matrix, and marking the image to be detected according to the target coordinates to obtain the target key point corresponding to each initial key point in the image to be detected.
[0009] In some embodiments, calculating the target human body position in the image to be detected based on the preset key points and the plurality of target coordinates includes: selecting a plurality of updated target coordinates from the plurality of target coordinates, and calculating the target human body position based on the plurality of updated target coordinates and a preset error value.
[0010] In some embodiments, the keypoint detection model is generated by: acquiring a keypoint detection network, multiple training images, and keypoint annotation results for each training image, wherein the keypoint annotation results include labeled keypoints in each training image and the labeled coordinates of the labeled keypoints in the corresponding training image; predicting each training image based on the keypoint detection network to obtain predicted keypoints and predicted coordinates corresponding to the labeled keypoints in each training image; calculating the network loss value of the keypoint detection network based on multiple labeled coordinates and the predicted coordinates corresponding to each labeled coordinate; and training the keypoint detection network based on the network loss value to obtain the keypoint detection model.
[0011] In some embodiments, the keypoint detection network includes a first detection network and a second detection network. Calculating the network loss value of the keypoint detection network based on the plurality of labeled coordinates and the predicted coordinates corresponding to each labeled coordinate includes: calculating a first loss value of the first detection network based on the coordinate distance between each labeled coordinate and its corresponding predicted coordinate; calculating a second loss value of the second detection network based on the average similarity between each labeled coordinate and its corresponding preset coordinate; and calculating the network loss value based on the first loss value and the second loss value.
[0012] On the other hand, this application provides a human body tracking device, the human body tracking device comprising: an acquisition unit for acquiring an image to be detected; a detection unit for performing human body detection on the image to be detected to obtain multiple detection key points in the image to be detected and detection coordinates of each detection key point; the acquisition unit is further configured to acquire preset coordinates of preset key points corresponding to each detection key point; a transformation unit for transforming the image to be detected according to a similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain a human body aligned image; an input unit for inputting the human body aligned image into a preset key point detection model if a human body is detected in the human body aligned image to obtain multiple initial key points in the human body aligned image and initial coordinates of each initial key point; and a calculation unit for calculating the target coordinates of a target key point corresponding to each initial key point in the image to be detected based on the similarity transformation matrix and the multiple initial coordinates, and calculating the target human body position in the image to be detected based on the preset key points and the multiple target coordinates.
[0013] On the other hand, this application provides an electronic device, the electronic device comprising: a memory storing at least one instruction; and a processor executing the at least one instruction to implement the human body tracking method.
[0014] On the other hand, this application provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the human body tracking method.
[0015] As can be seen from the above technical solution, the target human body position is the region corresponding to the human body in the image to be detected. When the keypoint detection network corresponding to the keypoint detection model is trained multiple times, updated target coordinates are generated after each training. Therefore, the target human body position can be adjusted based on the updated target coordinates, making the adjusted target human body position more accurate. Repeated training of the keypoint detection network with a more accurate target human body position can improve the accuracy of the keypoint detection model, thereby improving the detection accuracy of human body tracking. In addition, when detecting video images, the detection coordinates of the keypoints corresponding to the current frame image can be directly selected from the target coordinates of the target keypoints in the previous frame video image. Since it is not necessary to detect every frame image, the detection speed of human body tracking can be improved. Attached Figure Description
[0016] Figure 1 This is a structural diagram of an electronic device provided in an embodiment of this application.
[0017] Figure 2 This is a flowchart of a human body tracking method provided in an embodiment of this application.
[0018] Figure 3 This is a schematic diagram of the first pixel coordinate system provided in an embodiment of this application.
[0019] Figure 4 This is a flowchart of a method for calculating the similarity transformation matrix provided in an embodiment of this application.
[0020] Figure 5 This is a flowchart of a method for generating a key point detection model according to an embodiment of this application.
[0021] Figure 6 This is a flowchart of a human body detection method provided in an embodiment of this application.
[0022] Figure 7 This is a flowchart illustrating a method for calculating target key points and target coordinates according to an embodiment of this application.
[0023] Figure 8 This is a functional block diagram of a human body tracking device provided in an embodiment of this application. Detailed Implementation
[0024] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.
[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0026] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0027] The human body tracking method provided in this application can be applied to one or more electronic devices.
[0028] like Figure 1 The diagram shown is a structural diagram of an electronic device provided in an embodiment of this application. Figure 1 In the device 1, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and a computer program, such as a human tracking program, stored in the memory 12 and capable of running on the processor 13.
[0029] The electronic device 1 is a device that can automatically perform parameter value calculation and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to: microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0030] The electronic device 1 can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, in-vehicle device, etc.
[0031] The electronic device 1 may further include network devices and / or user devices. The network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers. Figure 1 This is merely an example of electronic device 1 and does not constitute a limitation on electronic device 1. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 1 may also include input / output devices, network access devices, buses, etc.
[0032] The network in which the electronic device 1 is located includes, but is not limited to: the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.
[0033] like Figure 2 The diagram shown is a flowchart of a human body tracking method according to an embodiment of this application. The order of the steps in the flowchart can be adjusted according to different needs, and some steps can be omitted. The method is executed by an electronic device, for example... Figure 1 Electronic device 1 shown.
[0034] S11, acquire the image to be detected.
[0035] In one embodiment, the image to be detected refers to an RGB image for which human key point detection is required.
[0036] In one embodiment, the electronic device may use an image generated by a shooting device after capturing a person as the image to be detected, or the electronic device may extract one or more frames from any video containing a person as the image to be detected. The shooting device includes, but is not limited to, an external camera or a webcam in an electronic device, etc.
[0037] S12, perform human body detection on the image to be detected to obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point.
[0038] In one embodiment, the electronic device performs human detection on the image to be detected to obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point, including: the electronic device uses a target detection algorithm to perform target detection on the image to be detected, selects the human body region containing the human body, and identifies the human body region to obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point.
[0039] The target detection algorithms include, but are not limited to, the OpenPose human pose estimation algorithm, the R-CNN target detection algorithm, the YOLO target detection algorithm, and the SSD target detection algorithm, etc. The methods for selecting and identifying the human body region using the target detection algorithms can refer to the relevant steps for anchor box selection and identification in related technologies.
[0040] In this embodiment, the target detection algorithm is used to perform preliminary human detection on the image to be detected, which provides a basis for the calculation of the similarity transformation matrix in the following text.
[0041] In one embodiment, the human body region is typically rectangular, so the electronic device can determine the coordinates of the vertices in the human body region as the initial human body position in the image to be detected. For example, the initial human body position may include a first initial human body coordinate of the upper left corner vertex of the human body region (the first initial coordinate includes the smallest initial x-coordinate and the smallest initial y-coordinate) and a second initial human body coordinate of the lower right corner vertex of the human body region (the second initial coordinate includes the largest initial x-coordinate and the largest initial y-coordinate); or, the initial human body position may include a third initial human body coordinate of the upper right corner vertex of the human body region (the third initial coordinate includes the largest initial x-coordinate and the smallest initial y-coordinate) and a fourth initial human body coordinate of the lower left corner vertex of the human body region (the fourth initial coordinate includes the smallest initial x-coordinate and the largest initial y-coordinate). The initial human body position may include the first initial human body coordinate of the upper left corner vertex of the human body region (the first initial coordinate includes the smallest initial x-coordinate and the smallest initial y-coordinate), the second initial human body coordinate of the lower right corner vertex of the human body region (the second initial coordinate includes the largest initial x-coordinate and the largest initial y-coordinate), the third initial human body coordinate of the upper right corner vertex of the human body region (the third initial coordinate includes the largest initial x-coordinate and the smallest initial y-coordinate), and the fourth initial human body coordinate of the lower left corner vertex of the human body region (the fourth initial coordinate includes the smallest initial x-coordinate and the largest initial y-coordinate).
[0042] S13, obtain the preset coordinates of the preset key points corresponding to each detection key point, and transform the image to be detected according to the similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain the human body aligned image.
[0043] In one embodiment, each preset key point refers to the key point corresponding to each detected key point in a standard human body image. The preset coordinates refer to the position coordinates of each preset key point in the first pixel coordinate system corresponding to the standard human body image. The number of preset key points is the same as the number of detected key points. The standard human body image can be a human body image of a person in a standard standing or lying posture. For example, if any detected key point is a nose, then the preset key point corresponding to any detected key point in the standard human body image is also a nose; or, if any detected key point is a left eye, then the preset key point corresponding to any detected key point in the standard human body image is also a left eye.
[0044] like Figure 3 The diagram shown is a schematic representation of a first pixel coordinate system provided in an embodiment of this application. The electronic device uses the pixel O in the first row and first column of the standard human body image as the coordinate system. uv Using the origin as the coordinate system, a first pixel coordinate system is constructed with the parallel line containing the first row of pixels as the u-axis and the vertical line containing the first column of pixels as the v-axis. The establishment of this first pixel coordinate system is merely an illustrative example and is not limited to this in practical applications.
[0045] In one embodiment, the human body alignment image is an image generated after performing transformation operations such as translation or rotation on the image to be detected.
[0046] In this embodiment, the electronic device transforms the image to be detected according to a plurality of similarity transformation matrices corresponding to a plurality of detection coordinates and a plurality of preset coordinates to obtain a human body aligned image, including:
[0047] The electronic device calculates the similarity transformation matrix based on the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates, and translates the image to be detected based on the similarity transformation matrix to obtain the human body aligned image.
[0048] Each detection coordinate includes a detection x-coordinate and a detection y-coordinate, and each preset coordinate also includes a preset x-coordinate and a preset y-coordinate. The detection vector is a vector composed of multiple detection x-coordinates and multiple detection coordinates, and the preset vector is a vector composed of multiple preset x-coordinates and multiple preset coordinates. The preset vector corresponds to the detection vector, and the arrangement order of each preset coordinate in the preset vector is the same as the arrangement order of the corresponding detection coordinates in the detection vector.
[0049] For example, if any two detection coordinates are A(50, 100) and B(100, 100), then the detection vector formed by A(50, 100) and B(100, 100) is [50100100100]. T The method of constructing the preset vector is basically the same as that of constructing the detection vector, so it will not be described again in this application.
[0050] In this embodiment, since the preset coordinates are the position coordinates of each preset key point in the first pixel coordinate system corresponding to the standard human body image, and the standard human body image is a human body image in a standard standing or lying posture, transforming the image to be detected according to the similarity transformation matrix can make the human body posture in the transformed human body alignment image a standard posture, thereby making it easier to detect the human body key points in the transformed human body alignment image.
[0051] In some embodiments, the detailed process of the electronic device calculating the similarity transformation matrix can be found in the following description. Figure 4 The process shown is described below.
[0052] S14, if a human body is detected in the human body alignment image, the human body alignment image is input into a preset key point detection model to obtain multiple initial key points in the human body alignment image and the initial coordinates of each initial key point.
[0053] In one embodiment, the keypoint detection model refers to a model capable of detecting human keypoints in an image containing a person. The keypoint detection model can be a single model (e.g., BlazePose, YOLOv7-Pose, or MoveNet, etc.) or a combination of multiple models. For example, the keypoint detection model can be an RLE+SimCC model combining Relation-Learned Embedding (RLE) and Similarity-based Coordinate Clustering (SimCC). The RLE model extracts global contextual information about the human pose in the human alignment image and predicts the human alignment image based on the extracted global contextual information to obtain preliminary predicted keypoints and their coordinates. The SimCC model is used to fine-tune the preliminary predicted keypoints and their coordinates. To better describe the human tracking algorithm in this application, the keypoint detection model will be described below using the RLE+SimCC model as an example.
[0054] In some embodiments, the keypoint detection model can be generated in a manner described below. Figure 5 The process shown is described below.
[0055] In some embodiments, the electronic device calculates the network loss value of the key point detection network based on a plurality of target coordinates and preset coordinates corresponding to each target coordinate, including:
[0056] The electronic device calculates a first loss value of the first detection network based on the coordinate distance between the plurality of target coordinates and the plurality of preset coordinates, and calculates a second loss value of the second detection network based on the average similarity between the plurality of target coordinates and the plurality of preset coordinates. Then, the electronic device calculates the network loss value based on the first loss value and the second loss value.
[0057] In some embodiments, current human keypoint detection technologies generate heatmaps using keypoint detection models and predict keypoints based on these heatmaps. However, these models require a large number of parameters to generate the heatmaps, resulting in slow keypoint detection speeds. To address this issue, this application employs a linear regression method to train the keypoint detection network, thereby reducing the number of parameters in the network. A comparison will be provided below with reference to Table 1.
[0058] Using the same backbone network (ShuffleNetv2) and the same input image, the second row and third column, and the second row and fourth column of Table 1 show the keypoint detection accuracy and detection speed of the RLE+SimCC model trained using linear regression, respectively. The third row and third column, and the third row and fourth column of Table 1 show the detection accuracy and detection speed of existing keypoint detection models for generating heatmaps, respectively.
[0059] Table 1 Detection speed and monitoring speed
[0060]
[0061] As can be seen from Table 1, the RLE+SimCC model trained by the linear regression method in this application has effectively improved the detection accuracy and speed of key points compared with the key point detection models that generate heatmaps in the prior art.
[0062] In this embodiment, if the keypoint detection model is an RLE+SimCC model, the RLE model supervises the predicted keypoints by treating each predicted keypoint and its coordinates as a distribution. The SimCC model, on the other hand, performs vector supervision by converting the pixel values corresponding to the predicted keypoints in the human alignment image into one-dimensional Gaussian vectors. RLE provides distribution supervision, while SimCC provides vector supervision. Training the keypoint detection network using these two hybrid supervision methods improves the detection accuracy. Furthermore, the use of linear regression to train the keypoint detection network reduces the number of parameters, thus increasing the generation speed of the keypoint detection model.
[0063] In some embodiments, the detailed steps of human detection performed by the electronic device on the human body aligned image can be referred to as follows: Figure 6 The process is illustrated below.
[0064] S15, calculate the target coordinates of the target key point corresponding to each initial key point in the image to be detected based on the similarity transformation matrix and the multiple initial coordinates, and calculate the target human body position in the image to be detected based on the preset key points and the multiple target coordinates.
[0065] In one embodiment, since the initial key points are located in the human body aligned image, which is an image generated by translating the image to be detected using the similarity transformation matrix, it is inaccurate to directly use the initial key points as human body key points in the image to be detected. Therefore, it is necessary to recalculate the initial coordinates of each initial key point using the similarity transformation matrix so that the calculated target key points and the target coordinates of the target key points are accurate human body key points on the image to be detected.
[0066] In some embodiments, the number of target key points can be multiple. For example, the multiple target key points include, but are not limited to: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right arm, left knee, right knee, left ankle, and right ankle, etc. Detailed calculation steps for the target key points and their target coordinates can be found in [reference needed]. Figure 7 The process is illustrated below.
[0067] In one embodiment, the electronic device calculates the target human body position in the image to be detected based on the preset key points and multiple target coordinates, including: the electronic device selects multiple updated target coordinates from the multiple target coordinates, and calculates the target human body position according to the multiple updated target coordinates and a preset error value, wherein the target human body position includes a first target human body coordinate, a second target human body coordinate, a third target human body coordinate, and a fourth target human body coordinate.
[0068] In this embodiment, the plurality of updated target coordinates include the largest target x-coordinate, the smallest target x-coordinate, the largest target y-coordinate, and the smallest target y-coordinate among the plurality of target coordinates. The preset error values include a first preset error value corresponding to the largest target x-coordinate, a second preset error value corresponding to the smallest target x-coordinate, a third preset error value corresponding to the largest target y-coordinate, and a fourth preset error value corresponding to the smallest target y-coordinate.
[0069] In some embodiments, the electronic device performs a preset operation between the largest target x-coordinate and the first preset error value to obtain the first target human body coordinates; performs the preset operation between the smallest target x-coordinate and the second preset error value to obtain the second target human body coordinates; performs the preset operation between the largest target y-coordinate and the third preset error value to obtain the third target human body coordinates; and performs the preset operation between the smallest target y-coordinate and the fourth preset error value to obtain the fourth target human body coordinates. The preset operations include, but are not limited to, addition, multiplication, division, and subtraction. The first, second, third, and fourth preset error values can all be set by the user, and this application does not impose any restrictions on them.
[0070] In some embodiments, after calculating the target human body position, the electronic device adjusts the initial human body position to the corresponding target human body position. For example, the largest initial x-coordinate corresponds to the largest target x-coordinate.
[0071] In this embodiment, the initial human body position is adjusted to the corresponding target human body position. Since the target human body position is calculated by a trained keypoint detection model, the target human body position is more accurate than the initial human body position.
[0072] As can be seen from the above technical solution, the target human body position is the region corresponding to the human body in the image to be detected. When the keypoint detection network corresponding to the keypoint detection model is trained multiple times, updated target coordinates are generated after each training. Therefore, the target human body position can be adjusted based on the updated target coordinates, making the adjusted target human body position more accurate. Repeated training of the keypoint detection network with a more accurate target human body position can improve the accuracy of the keypoint detection model, thereby improving the detection accuracy of human body tracking. In addition, when detecting video images, the detection coordinates of the keypoints corresponding to the current frame image can be directly selected from the target coordinates of the target keypoints in the previous frame video image. Since it is not necessary to detect every frame image, the detection speed of human body tracking can be improved.
[0073] like Figure 4 The diagram shown is a flowchart of a method for calculating a similarity transformation matrix according to an embodiment of this application, including the following steps:
[0074] S131, the electronic device constructs a vector equation based on the detection vector, the preset vector, the preset parameter matrix corresponding to the detection vector, and the parameter inverse matrix of the preset parameter matrix. The preset parameter matrix includes multiple preset parameters, which can be set independently; this application does not impose any restrictions on this. For example, the multiple preset parameters can be R1, R2, R3…Rn.
[0075] In some embodiments, the vector equation can be as shown in formula (1):
[0076] A = M -1 BM; (1)
[0077] Where A represents the preset vector, B represents the detection vector, and M represents the preset parameter matrix. -1 The parameter inverse matrix is the inverse of the preset parameter matrix. The method for obtaining the inverse matrix from the identity matrix can be found in related technologies.
[0078] S132, the electronic device uses the least squares method or the singular value decomposition (SVD) method to solve the vector equation to obtain the parameter value corresponding to each preset parameter in the preset parameter matrix.
[0079] In some embodiments, the method of solving the vector equation using the least squares method can refer to the application of the lsqr statement in MATLAB software in related technologies, and the method of solving the vector equation using the singular value decomposition method can refer to the application of the svd statement in MATLAB software in related technologies.
[0080] In other embodiments, the vector equations can be solved in other ways. For example, when the number of detection coordinates is small, multiple equations composed of multiple detection coordinates can be combined to solve the parameter value corresponding to each preset parameter in the preset parameter matrix. Alternatively, other solution statements can be used when solving by the least squares method or the singular value decomposition method. This application does not limit this.
[0081] S133, the electronic device replaces each preset parameter in the preset parameter matrix with the corresponding parameter value to obtain the similarity transformation matrix.
[0082] In some embodiments, the similarity transformation matrix includes the value corresponding to the translation operation of each detected abscissa and the value corresponding to the translation operation of each detected ordinate. To more clearly describe the composition of the similarity transformation matrix, a detailed explanation will be provided below in conjunction with formula (2).
[0083]
[0084] Where M represents the similarity transformation matrix, R 10 Equal to R 11 Equal, R 11 With R 11 T are opposites of each other. x T represents the value corresponding to the translation operation performed on each detected x-axis. y This represents the value corresponding to the translation operation performed on each detected vertical coordinate.
[0085] like Figure 5 The diagram shown is a flowchart of a method for generating a keypoint detection model according to an embodiment of this application, including the following steps:
[0086] S141, the electronic device acquires a key point detection network, multiple training images, and key point annotation results for each training image, wherein the key point annotation results include the annotated key points in each training image and the annotated coordinates of the annotated key points in the corresponding training image.
[0087] In some embodiments, the keypoint detection network can be the RLE+SimCC network corresponding to the RLE+SimCC model. The electronic device can obtain the multiple training images and the keypoint annotation results of each training image from a preset training dataset. The preset training dataset can be the MPII (MPII Human Pose Dataset), the MSCOCO dataset, or the PoseTrack dataset, etc. The training images in the above training dataset are images of a single person or multiple people, and multiple labeled keypoints and the labeled coordinates of each labeled keypoint are pre-annotated in each training image.
[0088] S142, the electronic device predicts each training image based on the key point detection network to obtain the predicted key points and the predicted coordinates corresponding to the labeled key points in each training image.
[0089] In some embodiments, if the keypoint detection model is the RLE+SimCC model, then the keypoint detection network includes a first detection network and a second detection network. The electronic device determines the RLE network corresponding to the RLE model as the first detection network and the SimCC network corresponding to the SimCC model as the second detection network.
[0090] S143, the electronic device calculates the network loss value of the key point detection network based on the multiple labeled coordinates and the predicted coordinates corresponding to each labeled coordinate, and trains the key point detection network based on the network loss value to obtain the key point detection model.
[0091] In some embodiments, the electronic device calculates the network loss value of the key point detection network based on the plurality of labeled coordinates and the predicted coordinates corresponding to each labeled coordinate, including: the electronic device calculates a first loss value of the first detection network based on the coordinate distance between each labeled coordinate and the corresponding predicted coordinate, calculates a second loss value of the second detection network based on the average similarity between each labeled coordinate and the corresponding preset coordinate, and calculates the network loss value based on the first loss value and the second loss value.
[0092] In one embodiment, the coordinate distance can be the Euclidean distance between each labeled coordinate and its corresponding predicted coordinate. The electronic device can determine the first loss value as the ratio between the sum of the distances and the number of predictions, wherein the sum of the distances is the sum of the multiple coordinate distances corresponding to the multiple predicted coordinates, and the number of predictions is the number of the multiple predicted coordinates.
[0093] In one embodiment, the electronic device can determine the second loss value as the ratio between the sum of multiple similarities and the number of similarities of the multiple similarities, wherein each similarity is the cosine of the angle between each pair of predicted vectors and the corresponding labeled vector, each pair of predicted vectors is the unit vector corresponding to the relative position vector between any two predicted coordinates, and each pair of labeled vectors is the unit vector corresponding to the relative position vector between any two labeled coordinates. The electronic device determines the network loss value as the weighted average between the first loss value and the second loss value.
[0094] The relative position vector between any two predicted coordinates is a vector formed by the two predicted coordinates. For example, if the two predicted coordinates are C(5, 1) and C(1, 10), then the detection vector formed by C(5, 1) and D(1, 10) is
[51110] . T The electronic device normalizes the relative position vector to obtain a unit vector corresponding to the relative position vector between any two predicted coordinates.
[0095] like Figure 6 The diagram shown is a flowchart of a human body detection method provided in an embodiment of this application, including the following steps:
[0096] S144, the electronic device extracts features from the human body alignment image to obtain a feature map.
[0097] In some embodiments, the electronic device may employ convolutional layers and pooling layers in a prediction algorithm to extract features from the human body alignment image to obtain the feature map. This application does not limit the prediction algorithm. For example, the prediction algorithm may be a decision tree algorithm, a support vector machine algorithm, a linear regression algorithm, or a random forest algorithm, etc. In addition to the convolutional layers, the prediction algorithm may also include fully connected layers and activation layers, etc.
[0098] S145, the electronic device performs human body prediction on the feature map to obtain the predicted probability that the human body aligned image contains a human body.
[0099] In some embodiments, the electronic device classifies the feature map based on the fully connected layer to obtain a predicted probability that the image to be detected contains a human body. The method of classifying the feature map using the fully connected layer can refer to related techniques that classify using an activation function. The activation function can be a softmax function.
[0100] S146, if the predicted probability is greater than or equal to a preset threshold, the electronic device determines that the human body alignment image contains a human body.
[0101] In some embodiments, the preset threshold can be set independently, and this application does not impose any restrictions on it. For example, the preset threshold can be 0.9 or 0.8.
[0102] S147, if the predicted probability is less than the preset threshold, the electronic device determines that the human body alignment image does not contain a human body.
[0103] In some embodiments, when the predicted probability is less than the preset threshold, the electronic device returns to step S14 and continues to perform human detection on the human body alignment image until a human body is detected in the human body alignment image or the number of human body detections exceeds a preset configuration value. The preset configuration value can be set arbitrarily, and this application does not impose any restrictions on it. For example, the preset configuration value can be 3 times.
[0104] like Figure 7 The diagram shown is a flowchart of a method for calculating target key points and target coordinates according to an embodiment of this application.
[0105] S151, the electronic device obtains the transformation inverse matrix of the similarity transformation matrix, and calculates the target coordinates corresponding to each initial coordinate based on each initial coordinate and the transformation inverse matrix.
[0106] In one embodiment of this application, the inverse transformation matrix is the inverse of the similarity transformation matrix. The calculation process of the target coordinates and the calculation process of the similarity transformation matrix are inverse processes, therefore, this application will not repeat the description.
[0107] S152, the electronic device marks the image to be detected according to the target coordinates to obtain the target key point corresponding to each initial key point in the image to be detected.
[0108] In one embodiment of this application, there are multiple target coordinates, and the electronic device determines the point of each target coordinate on the second pixel coordinate system corresponding to the image to be detected as a target key point corresponding to each target coordinate. It is understood that the process of establishing the second pixel coordinate system is basically the same as the process of establishing the first pixel coordinate system, therefore, this application will not repeat the description.
[0109] like Figure 8 The diagram shown is a functional block diagram of a human body tracking device according to an embodiment of this application. The human body tracking device 11 includes an acquisition unit 110, a detection unit 111, a transformation unit 112, an input unit 113, and a calculation unit 114. The module / unit referred to in this application refers to a module / unit capable of being... Figure 1 The processor 13 in the middle acquires a series of computer-readable instruction segments that are capable of performing a fixed function, and these segments are stored in Figure 1The memory 12 is used for this purpose. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.
[0110] The acquisition unit 110 is used to acquire the image to be detected.
[0111] The detection unit 111 is used to perform human body detection on the image to be detected, and obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point.
[0112] The acquisition unit 110 is also used to acquire the preset coordinates of the preset key points corresponding to each detection key point.
[0113] The transformation unit 112 is used to transform the image to be detected according to the similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain a human body aligned image.
[0114] The input unit 113 is used to input the human body alignment image into a preset key point detection model if a human body is detected in the human body alignment image, so as to obtain multiple initial key points in the human body alignment image and the initial coordinates of each initial key point.
[0115] The calculation unit 114 is used to calculate the target coordinates of each initial key point in the target key point corresponding to the image to be detected based on the similarity transformation matrix and multiple initial coordinates, and to calculate the target human body position in the image to be detected based on the preset key points and multiple target coordinates.
[0116] In one embodiment of this application, the detection unit 111 is further configured to: extract features from the human body alignment image to obtain a feature map; predict the human body in the feature map to obtain a predicted probability that the human body alignment image contains a human body; if the predicted probability is greater than or equal to a preset threshold, determine that the human body alignment image contains a human body, or if the predicted probability is less than the preset threshold, determine that the human body alignment image does not contain a human body.
[0117] In one embodiment of this application, the transformation unit 112 is further configured to: calculate the similarity transformation matrix based on the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates; and translate the image to be detected based on the similarity transformation matrix to obtain the human body aligned image.
[0118] In one embodiment of this application, the calculation unit 114 is further configured to: construct a vector equation based on the detection vector, the preset vector, the preset parameter matrix corresponding to the detection vector, and the parameter inverse matrix of the preset parameter matrix, wherein the preset parameter matrix includes multiple preset parameters; solve the vector equation using the least squares method or the singular value decomposition method to obtain the parameter value corresponding to each preset parameter in the preset parameter matrix; and replace each preset parameter in the preset parameter matrix with the corresponding parameter value to obtain the similarity transformation matrix.
[0119] In one embodiment of this application, the calculation unit 114 is further configured to: obtain the transformation inverse matrix of the similarity transformation matrix, and calculate the target coordinates corresponding to each initial coordinate based on each initial coordinate and the transformation inverse matrix; and mark the target coordinates in the image to be detected to obtain the target key point corresponding to each initial key point in the image to be detected.
[0120] In one embodiment of this application, the calculation unit 114 is further configured to: select multiple updated target coordinates from the plurality of target coordinates; and calculate the position of the target human body based on the plurality of updated target coordinates and a preset error value.
[0121] In one embodiment of this application, the computing unit 114 is further configured to: acquire a keypoint detection network, multiple training images, and keypoint annotation results for each training image, wherein the keypoint annotation results include labeled keypoints in each training image and labeled coordinates of the labeled keypoints in the corresponding training image; predict each training image based on the keypoint detection network to obtain predicted keypoints corresponding to the labeled keypoints in each training image and the predicted coordinates; calculate the network loss value of the keypoint detection network based on multiple labeled coordinates and the predicted coordinates corresponding to each labeled coordinate, and train the keypoint detection network based on the network loss value to obtain the keypoint detection model.
[0122] In one embodiment of this application, the key point detection network includes a first detection network and a second detection network. The calculation unit 114 is further configured to: calculate a first loss value of the first detection network based on the coordinate distance between each labeled coordinate and the corresponding predicted coordinate, and calculate a second loss value of the second detection network based on the average similarity between each labeled coordinate and the corresponding preset coordinate; and calculate the network loss value based on the first loss value and the second loss value.
[0123] As can be seen from the above technical solution, the target human body position is the region corresponding to the human body in the image to be detected. When the keypoint detection network corresponding to the keypoint detection model is trained multiple times, updated target coordinates are generated after each training. Therefore, the target human body position can be adjusted based on the updated target coordinates, making the adjusted target human body position more accurate. Repeated training of the keypoint detection network with a more accurate target human body position can improve the accuracy of the keypoint detection model, thereby improving the detection accuracy of human body tracking. In addition, when detecting video images, the detection coordinates of the keypoints corresponding to the current frame image can be directly selected from the target coordinates of the target keypoints in the previous frame video image. Since it is not necessary to detect every frame image, the detection speed of human body tracking can be improved.
[0124] In one embodiment, continuing from the preceding text... Figure 1 In the description of the electronic device, processor 13 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. Processor 13 is the computing core and control center of electronic device 1, connecting various parts of electronic device 1 through various interfaces and lines, and obtaining the operating system of electronic device 1 and various installed applications and program code.
[0125] Processor 13 acquires the operating system and various installed applications of electronic device 1. Processor 13 acquires these applications to implement the steps described in the various human body tracking method embodiments above, for example... Figure 2 , Figure 4 , Figure 5 , Figure 6 as well as Figure 7 The steps are shown.
[0126] For example, a computer program may be divided into one or more modules / units, such as an acceleration unit. One or more modules / units are stored in memory 12 and retrieved by processor 13 to complete this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the process of retrieving the computer program in electronic device 1.
[0127] The memory 12 can be used to store computer programs and / or modules. The processor 13 implements various functions of the electronic device 1 by running or retrieving the computer programs and / or modules stored in the memory 12, and by calling the data stored in the memory 12. The memory 12 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 12 may include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.
[0128] The memory 12 can be the external memory and / or internal memory of the electronic device 1. Furthermore, the memory 12 can be a memory in physical form, such as a memory stick, a TF card (Trans-flash Card), etc.
[0129] If the modules / units integrated in electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is acquired by a processor, it can implement the steps of the various method embodiments described above.
[0130] Computer programs include computer program code, which can be in the form of source code, object code, accessible files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).
[0131] Combination Figure 2The memory 12 in the electronic device 1 stores multiple instructions to implement a human body tracking method. The processor 13 can acquire multiple instructions to implement: acquiring an image to be detected; performing human body detection on the image to be detected to obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point; acquiring the preset coordinates of preset key points corresponding to each detection key point, and transforming the image to be detected according to the similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain a human body aligned image; if a human body is detected in the human body aligned image, inputting the human body aligned image into a preset key point detection model to obtain multiple initial key points in the human body aligned image and the initial coordinates of each initial key point; calculating the target coordinates of the target key point corresponding to each initial key point in the image to be detected based on the similarity transformation matrix and the multiple initial coordinates, and calculating the target human body position in the image to be detected based on the preset key points and the multiple target coordinates.
[0132] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.
[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0134] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0135] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0136] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.
[0137] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.
[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.
Claims
1. A human body tracking method, characterized in that, The method includes: Acquire the image to be detected; Human body detection is performed on the image to be detected to obtain multiple detection key points in the image to be detected and the detection coordinates of each detection key point; Obtain the preset coordinates of the preset key points corresponding to each detection key point, and transform the image to be detected according to the similarity transformation matrix corresponding to the multiple detection coordinates and the multiple preset coordinates to obtain the human body aligned image; A keypoint detection network, multiple training images, and keypoint annotation results for each training image are obtained. The keypoint annotation results include the annotated keypoints in each training image and the annotated coordinates of the annotated keypoints in the corresponding training image. The keypoint detection network is used to predict the keypoints and their corresponding coordinates in each training image. Based on the multiple annotated coordinates and the predicted coordinates corresponding to each annotated coordinate, the network loss value of the keypoint detection network is calculated. The keypoint detection network is then trained based on the network loss value to obtain a keypoint detection model. The keypoint detection network includes a first detection network and a second detection network. The step of calculating the network loss value of the key point detection network based on multiple labeled coordinates and the predicted coordinates corresponding to each labeled coordinate includes: calculating the first loss value of the first detection network based on the coordinate distance between each labeled coordinate and the corresponding predicted coordinate, calculating the second loss value of the second detection network based on the average similarity between each labeled coordinate and the corresponding preset coordinate, and calculating the network loss value based on the first loss value and the second loss value. If a human body is detected in the human body alignment image, the human body alignment image is input into the keypoint detection model to obtain multiple initial keypoints in the human body alignment image and the initial coordinates of each initial keypoint. This includes: extracting global context information of the human body pose in the human body alignment image using a first detection network in the keypoint detection model, and making predictions based on the global context information to obtain predicted keypoints and the coordinates of the predicted keypoints; adjusting the coordinates of the predicted keypoints using a second detection network in the keypoint detection model to obtain the multiple initial keypoints and the initial coordinates of each initial keypoint. Based on the similarity transformation matrix and multiple initial coordinates, the target coordinates of the target key point corresponding to each initial key point in the image to be detected are calculated, and the target human body position in the image to be detected is calculated based on the preset key points and multiple target coordinates.
2. The human body tracking method as described in claim 1, characterized in that, The human detection performed on the human body aligned image includes: Feature extraction is performed on the human body alignment image to obtain a feature map; Human body prediction is performed on the feature map to obtain the predicted probability that the human body aligned image contains a human body; If the predicted probability is greater than or equal to a preset threshold, it is determined that the human body alignment image contains a human body; or If the predicted probability is less than the preset threshold, it is determined that the human body alignment image does not contain a human body.
3. The human body tracking method as described in claim 1, characterized in that, The step of transforming the image to be detected according to a plurality of detection coordinates and a plurality of preset coordinates to obtain a human body aligned image includes: The similarity transformation matrix is calculated based on the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates. The image to be detected is translated according to the similarity transformation matrix to obtain the human body aligned image.
4. The human body tracking method as described in claim 3, characterized in that, The step of calculating the similarity transformation matrix based on the detection vectors corresponding to the plurality of detection coordinates and the preset vectors corresponding to the plurality of preset coordinates includes: A vector equation is constructed based on the detection vector, the preset vector, the preset parameter matrix corresponding to the detection vector, and the parameter inverse matrix of the preset parameter matrix, wherein the preset parameter matrix includes multiple preset parameters; The vector equation is solved using the least squares method or singular value decomposition method to obtain the parameter value corresponding to each preset parameter in the preset parameter matrix; In the preset parameter matrix, each preset parameter is replaced with its corresponding parameter value to obtain the similarity transformation matrix.
5. The human body tracking method as described in claim 1, characterized in that, The step of calculating the target coordinates of each initial key point in the target key point of the image to be detected based on the similarity transformation matrix and multiple initial coordinates includes: Obtain the transformation inverse matrix of the similarity transformation matrix, and calculate the target coordinates corresponding to each initial coordinate based on each initial coordinate and the transformation inverse matrix; The target coordinates are used to mark the image to be detected, thereby obtaining the target key point corresponding to each initial key point in the image to be detected.
6. The human body tracking method as described in claim 1, characterized in that, The step of calculating the target human body position in the image to be detected based on the preset key points and multiple target coordinates includes: Select multiple updated target coordinates from the plurality of target coordinates; The position of the target human body is calculated based on the multiple updated target coordinates and the preset error value.
7. An electronic device, characterized in that, The electronic device includes: Memory, storing at least one instruction; and The processor executes the at least one instruction to implement the human body tracking method as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in an electronic device to implement the human body tracking method as described in any one of claims 1 to 6.