Gait recognition method and model training method and device thereof, terminal and storage medium

Abstract

The application provides a gait recognition method and a gait recognition model training method, device, terminal and storage medium. The gait recognition model training method comprises the following steps: acquiring a training video stream, the training video stream having a labeled category of a target object; performing feature extraction on a fused image sequence of the training video stream by using a teacher model to obtain feature information of the training video stream; determining a first predicted category and a predicted feature of the target object based on a mask image sequence corresponding to the training video stream by using a gait recognition model; and iteratively training the gait recognition model based on a first error value between the labeled category and the first predicted category of the same training video stream and a second error value between the feature information and the predicted feature. The gait recognition model based on the mask image sequence of the single-channel data can also obtain the same or similar target category and target feature based on the fused image sequence, thereby improving the recognition accuracy and shortening the time consumption.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition technology, and in particular to a gait recognition method and its model training method, device, terminal, and storage medium. Background Technology

[0002] Gait recognition, as an emerging identity recognition technology, has many advantages such as being non-contact and long-distance, and is increasingly becoming a hot topic of research.

[0003] Most current mainstream gait recognition algorithms use statistical or deep learning models to learn parameters from pedestrian silhouette sequences. Based on this, a feature extractor is derived, and then feature matching is performed using the extracted features to identify the person. While pedestrian silhouettes can effectively remove interference from the color information of clothing and belongings, the differences in the silhouettes of different clothing and belongings can make the differences between silhouettes of the same person before and after clothing changes greater than the differences between silhouettes of different people. This undoubtedly affects the accuracy of gait recognition. Summary of the Invention

[0004] The main technical problem solved by this invention is to provide a gait recognition method and its model training method, device, terminal, and storage medium, which solves the problems of low accuracy and long processing time in the prior art.

[0005] To address the aforementioned technical problems, the first technical solution adopted by this invention is: providing a gait recognition model training method, which includes: acquiring a training video stream, the training video stream containing multiple frames of sample images with the same target object, the training video stream being associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has labeled categories of the target object; feature extraction is performed on the fused image sequence of the training video stream using a teacher model to obtain feature information of the training video stream; the gait recognition model determines a first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream; the teacher model is a model that has been trained and performs the same task as the gait recognition model; and the gait recognition model is iteratively trained based on a first error value between the labeled category and the first predicted category corresponding to the same training video stream and a second error value between the feature information and the predicted features.

[0006] The process of acquiring the training video stream includes: segmenting each sample image to obtain a mask image of the sample image; wherein the mask image is single-channel data; and arranging the mask images corresponding to all sample images in the training video stream in temporal order to generate a mask image sequence corresponding to the training video stream.

[0007] The acquisition of the training video stream also includes: performing keypoint detection on each sample image to obtain the keypoint information of the corresponding part of the sample image; generating a keypoint connection structure diagram corresponding to the sample image based on the keypoint information of the part of the sample image; generating a fused image of the sample images based on the mask image, keypoint information of the part, and keypoint connection structure diagram corresponding to each sample image; the fused image is multi-channel data; and arranging the fused images corresponding to all sample images in the training video stream in temporal order to generate a fused image sequence corresponding to the training video stream.

[0008] The fused image consists of three channels. Based on the mask image, key point information, and key point connection structure diagram corresponding to each sample image, a fused image of the sample images is generated, including: copying and stitching the single-channel mask image to obtain a three-channel mask image; copying and stitching the single-channel key point information to obtain three-channel key point information; and fusing the three-channel mask image, three-channel key point information, and key point connection structure diagram corresponding to the same sample image to generate the fused image of the sample images.

[0009] The key point information of the part includes the key points of the part, as well as the category and location of the key points of the part; based on the key point information of the part of the sample image, a key point connection structure diagram corresponding to the sample image is generated, including: based on the category and location of the key points of the part, connecting the key points of the part with different types of line segments to obtain the key point connection structure diagram corresponding to the sample image.

[0010] The categories of key points include nose tip, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and / or right ankle. Based on the category and location of the key points, the key points are connected using line segments of different types to obtain the key point connection structure diagram corresponding to the sample image. This includes: in response to one key point being classified as left ankle and another as left knee, and / or one key point being classified as right ankle and another as right knee, the two key points are connected using a line segment of the first color; and / or; in response to one key point being classified as left hip and another as left knee, and / or one key point being classified as right hip and another as right knee, the two key points are connected using a line segment of the second color; and / or; in response to one key point being classified as left hip and another as left shoulder, and / or one key point being classified as right hip and another as right shoulder, the line segment of the third color is used. A line segment of the fourth color connects two key points; and / or; in response to one key point being classified as left elbow and the other as left shoulder, and / or one key point being classified as right elbow and the other as right shoulder, the two key points are connected by a line segment of the fourth color; and / or; in response to one key point being classified as left elbow and the other as left wrist, and / or one key point being classified as right elbow and the other as right wrist, the two key points are connected by a line segment of the fifth color; and / or; in response to one key point being classified as right shoulder and the other as left shoulder, and / or one key point being classified as right hip and the other as left hip, the two key points are connected by a line segment of the sixth color; and / or; in response to one key point being classified as nose tip and the other two key points being classified as left shoulder and right shoulder respectively, the midpoint of the line segment corresponding to the nose tip is connected to the midpoint of the line segment corresponding to the left shoulder and right shoulder using a line segment of the seventh color.

[0011] The training method for the teacher model includes: using the teacher model to obtain the second predicted category of the target object based on the fused image sequence of the training video stream; and iteratively training the teacher model based on the third error value between the second predicted category and the labeled category corresponding to the training video stream.

[0012] The teacher model includes a feature extraction module and a discrimination module. Based on the fused image sequence of the training video stream, the teacher model obtains the second predicted category of the target object. This includes: using the feature extraction module to extract features from the fused image sequence to obtain the feature vector of the target object, which contains temporal information; using the discrimination module to calculate the similarity between the feature vector and each preset feature vector; and taking the category of the preset vector with the highest similarity as the second predicted category of the target object.

[0013] To solve the above-mentioned technical problems, the second technical solution adopted by the present invention is: to provide a gait recognition method, the gait recognition method comprising: acquiring a video stream, the video stream comprising multiple video frames containing the same target; segmenting each video frame to obtain a mask image sequence of the video stream; and using a gait recognition model to determine the identity category of the target based on the mask image sequence corresponding to the video stream; wherein the gait recognition model is trained by the above-mentioned gait recognition model training method.

[0014] To address the aforementioned technical problems, the third technical solution adopted by this invention is: providing a gait recognition model training device, comprising: an acquisition module for acquiring a training video stream, the training video stream containing multiple frames of sample images with the same target object, the training video stream being associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has labeled categories of the target object; a supervision module for extracting features from the fused image sequence of the training video stream using a teacher model to obtain feature information of the training video stream; a processing module for determining a first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream using a gait recognition model; the teacher model is a model that has been trained and performs the same task as the gait recognition model; and a training module for iteratively training the gait recognition model based on a first error value between the labeled category and the first predicted category corresponding to the same training video stream and a second error value between the feature information and the predicted features.

[0015] To solve the above-mentioned technical problems, the fourth technical solution adopted by the present invention is: to provide a gait recognition device, the gait recognition device comprising: an information acquisition module for acquiring a video stream, the video stream comprising multiple video frames containing the same target; a preprocessing module for segmenting each video frame to obtain a mask image sequence of the video stream; and a determination module for determining the identity category of the target based on the mask image sequence corresponding to the video stream using a gait recognition model; wherein the gait recognition model is trained by the above-mentioned gait recognition model training method.

[0016] To solve the above-mentioned technical problems, the fifth technical solution adopted by the present invention is: to provide a terminal, the terminal including a memory, a processor and a computer program stored in the memory and running on the processor, the processor being used to execute program data to implement the above-mentioned gait recognition model training method or the steps of the above-mentioned gait recognition method.

[0017] To solve the above-mentioned technical problems, the sixth technical solution adopted by the present invention is: to provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, to implement the above-mentioned gait recognition model training method or the steps of the above-mentioned gait recognition method.

[0018] The beneficial effects of this invention are as follows: Unlike existing technologies, it provides a gait recognition method and its model training method, apparatus, terminal, and storage medium. The gait recognition model training method includes: acquiring a training video stream, which contains multiple frames of sample images with the same target object; the training video stream is associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has labeled categories of the target object; feature extraction is performed on the fused image sequence of the training video stream using a teacher model to obtain feature information of the training video stream; the gait recognition model determines a first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream; the teacher model is a model that has been trained and performs the same task as the gait recognition model; and the gait recognition model is iteratively trained based on a first error value between the labeled category and the first predicted category corresponding to the same training video stream and a second error value between the feature information and the predicted features. This application generates feature information of the training video stream based on the fused image sequence of the training video stream using a teacher model. This feature information is then used as supervision information. A gait recognition model is then used to determine the first predicted category and predicted features of the target object based on the masked image sequence of the training video stream. The gait recognition model is trained using the error values ​​between the predicted features and the supervision information, and the error values ​​between the first predicted category and the labeled category. This allows the gait recognition model to obtain the same or similar target categories and target features based on the masked image sequence of single-channel data or the fused image sequence, thereby improving the recognition accuracy of the gait recognition model and reducing the computational load, thus shortening the recognition process time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the gait recognition model training method provided by the present invention;

[0021] Figure 2 This is a flowchart illustrating a specific embodiment of the gait recognition model training method provided by the present invention;

[0022] Figure 3 This is a schematic block diagram of an embodiment of the gait recognition model or teacher model provided by the present invention;

[0023] Figure 4 This is a flowchart illustrating an embodiment of the gait recognition method provided by the present invention;

[0024] Figure 5 This is a schematic diagram of the framework of an embodiment of the gait recognition model training device provided by the present invention;

[0025] Figure 6 This is a schematic diagram of the framework of an embodiment of the gait recognition device provided by the present invention;

[0026] Figure 7 This is a schematic diagram of the framework of an embodiment of the terminal provided in this application;

[0027] Figure 8 A schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0029] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0030] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0031] To enable those skilled in the art to better understand the technical solution of the present invention, the following describes in further detail a gait recognition model training method and a gait recognition method provided by the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0032] Please see Figure 1 , Figure 1This is a flowchart illustrating an embodiment of the gait recognition model training method provided by the present invention. This embodiment provides a gait recognition model training method, which includes the following steps.

[0033] S11: Obtain the training video stream.

[0034] Specifically, the training video stream contains multiple frames of sample images with the same target object. The training video stream is associated with a corresponding mask image sequence and a fused image sequence. The fused image sequence is multi-channel data, and the mask image sequence is single-channel data. The training video stream has labeled categories of the target object.

[0035] In one embodiment, a training video stream is acquired, which contains multiple frames of sample images with the same target object; the training video stream has labeled categories of the target object; each sample image is segmented to obtain a mask image of the sample image; wherein the mask image is single-channel data; the mask images corresponding to all sample images in the training video stream are arranged in time sequence to generate a mask image sequence corresponding to the training video stream.

[0036] In one embodiment, keypoint detection is performed on each sample image to obtain keypoint information of the corresponding body part; based on the keypoint information of the body part of the sample image, a keypoint connection structure diagram corresponding to the sample image is generated; according to the mask image, body part keypoint information, and keypoint connection structure diagram corresponding to each sample image, a fused image of the sample images is generated; the fused image is multi-channel data; the fused images corresponding to all sample images in the training video stream are arranged in temporal order to generate a fused image sequence corresponding to the training video stream. The fused image is three-channel data.

[0037] In one specific embodiment, a single-channel mask image is copied and stitched together to obtain a three-channel mask image; the single-channel key point information is copied and stitched together to obtain three-channel key point information; the three-channel mask image, the three-channel key point information, and the key point connection structure diagram corresponding to the same sample image are fused to generate a fused image of the sample image.

[0038] In one specific embodiment, the key point information of the part includes the key points of the part as well as the category and location of the key points of the part; based on the category and location of the key points of the part, the key points of the part are connected by line segments of different types to obtain the key point connection structure diagram corresponding to the sample image.

[0039] In one specific embodiment, the categories of key points of the body part include the tip of the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle and / or right ankle.

[0040] In response to a key point being categorized as left ankle and another key point as left knee, and / or a key point being categorized as right ankle and another key point as right knee, the two key points are connected by a line segment of the first color.

[0041] In response to a key point being categorized as left hip and another key point as left knee, and / or a key point being categorized as right hip and another key point as right knee, the two key points are connected by a line segment of the second color.

[0042] In response to a key point being categorized as left hip and another key point as left shoulder, and / or a key point being categorized as right hip and another key point as right shoulder, the two key points are connected by a line segment of the third color.

[0043] In response to a key point being categorized as left elbow and another key point as left shoulder, and / or a key point being categorized as right elbow and another key point as right shoulder, the two key points are connected by a line segment of the fourth color.

[0044] In response to a key point being categorized as left elbow and another key point as left wrist, and / or a key point being categorized as right elbow and another key point as right wrist, the two key points are connected by a line segment of the fifth color.

[0045] In response to a key point being categorized as right shoulder and another key point as left shoulder, and / or a key point being categorized as right hip and another key point as left hip, the two key points are connected by a line segment of the sixth color.

[0046] If one key point is categorized as the tip of the nose and the other two key points are categorized as the left shoulder and the right shoulder, then the midpoints of the line segments corresponding to the tip of the nose and the left and right shoulders are connected by the line segments of the seventh color.

[0047] S12: The teacher model extracts features from the fused image sequence of the training video stream to obtain the feature information of the training video stream.

[0048] Specifically, a teacher model is used to obtain the second predicted category of the target object based on the fused image sequence of the training video stream; the teacher model is then iteratively trained based on the third error value between the second predicted category and the labeled category corresponding to the training video stream.

[0049] In one embodiment, the teacher model includes a feature extraction module and a discrimination module. The feature extraction module extracts features from the fused image sequence to obtain feature vectors of the target object, which contain temporal information. The discrimination module calculates the similarity between the feature vectors and each preset feature vector. The category of the preset vector corresponding to the highest similarity is used as the second predicted category of the target object.

[0050] S13: Based on the masked image sequence corresponding to the training video stream, the gait recognition model determines the first predicted category and predicted features of the target object; the teacher model is a model that has been trained and performs the same task as the gait recognition model.

[0051] S14: Iteratively train the gait recognition model based on the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted features.

[0052] The gait recognition model training method provided in this embodiment includes: acquiring a training video stream, which contains multiple frames of sample images with the same target object; the training video stream is associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has labeled categories of the target object; feature extraction is performed on the fused image sequence of the training video stream using a teacher model to obtain feature information of the training video stream; the gait recognition model determines the first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream; the teacher model is a model that has been trained and performs the same task as the gait recognition model; and the gait recognition model is iteratively trained based on the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted features. This application generates feature information of the training video stream based on the fused image sequence of the training video stream using a teacher model. This feature information is then used as supervision information. A gait recognition model is then used to determine the first predicted category and predicted features of the target object based on the masked image sequence of the training video stream. The gait recognition model is trained using the error values ​​between the predicted features and the supervision information, and the error values ​​between the first predicted category and the labeled category. This allows the gait recognition model to obtain the same or similar target categories and target features based on the masked image sequence of single-channel data or the fused image sequence, thereby improving the recognition accuracy of the gait recognition model and reducing the computational load, thus shortening the recognition process time.

[0053] Please see Figure 2 , Figure 2 This is a flowchart illustrating a specific embodiment of the gait recognition model training method provided by the present invention. This embodiment provides a gait recognition model training method, which includes the following steps.

[0054] S201: Obtain the training video stream.

[0055] Specifically, multiple training video streams are acquired, each containing multiple consecutive sample images of the same target object. In other words, each training video stream corresponds to one target object. Each training video stream is associated with a labeled category for the target object. The labeled category includes the target object's identity information. In this embodiment, the target object can be a pedestrian or another animal.

[0056] In one embodiment, the sample images contained in the training video stream are divided into consecutive RGB image sequences according to time order. All sample images contained in the same training video stream are images of the same target object taken from different shooting angles.

[0057] S202: Perform segmentation processing on each sample image to obtain the mask image of the sample image.

[0058] Specifically, background subtraction or a trained segmentation model is used to segment each sample image to obtain the pedestrian contour map corresponding to each sample image.

[0059] In one embodiment, target objects in a sample image are detected, and the sample image is binarized. Specifically, pixels at the location of the target object in the sample image are assigned a value of 1, while pixels at the location of the background region are assigned a value of 0, resulting in a mask image corresponding to the sample image. In this embodiment, the mask image is specifically a pedestrian contour map. In this embodiment, the mask image is single-channel data.

[0060] This step allows us to obtain the mask image corresponding to each sample image, and then arrange the mask images corresponding to all sample images in the training video stream in chronological order to generate a sequence of mask images corresponding to the training video stream.

[0061] S203: Perform key point detection on each sample image to obtain key point information of the corresponding part of the sample image.

[0062] Specifically, a joint point estimation algorithm is used to detect joint points in each sample image to obtain the key point information of the target object in each sample image. This key point information includes the key point itself, its category, and its location. The categories of key points include nose tip, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and / or right ankle. In other embodiments, other categories of joint points can be added as needed. Key points can be in the form of solid circles or Gaussian points.

[0063] This step allows us to obtain key point information for the corresponding parts of each sample image, and the key point information for each sample image corresponds one-to-one with the mask image of the sample image.

[0064] S204: Based on the key point information of the sample image, generate a key point connection structure diagram corresponding to the sample image.

[0065] Specifically, based on the category and location of key points in a body part, key points are connected using line segments of different types to obtain a key point connection structure diagram corresponding to the sample image. Since the target object in this embodiment is a pedestrian, the key point information includes the joints of the body part, the category of the joints, and the coordinate position of the joints in the sample image. Based on the category of the pedestrian's joints and their coordinate positions in the sample image, the joints are connected using line segments of different colors, thereby constructing a human skeletal structure diagram of the pedestrian in the sample image.

[0066] In one embodiment, the key point categories include the tip of the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and / or right ankle.

[0067] In one embodiment, in response to one site keypoint being classified as left ankle and another site keypoint as left knee, and / or one site keypoint being classified as right ankle and another site keypoint as right knee, the two site keypoints are connected by a line segment of the first color.

[0068] In one embodiment, in response to one key point being categorized as left hip and another key point as left knee, and / or one key point being categorized as right hip and another key point as right knee, the two key points are connected by a line segment of the second color.

[0069] In one embodiment, in response to one key point being categorized as left hip and another key point as left shoulder, and / or one key point being categorized as right hip and another key point as right shoulder, the two key points are connected by a line segment of the third color.

[0070] In one embodiment, in response to one location keypoint being categorized as left elbow and another location keypoint as left shoulder, and / or one location keypoint being categorized as right elbow and another location keypoint as right shoulder, the two location keypoints are connected by a line segment of the fourth color.

[0071] In one embodiment, in response to one location key point being classified as left elbow and another location key point as left wrist, and / or one location key point being classified as right elbow and another location key point as right wrist, the two location key points are connected by a line segment of the fifth color.

[0072] In one embodiment, in response to one key feature being categorized as right shoulder and another as left shoulder, and / or one key feature being categorized as right hip and another as left hip, the two key features are connected by a line segment of the sixth color.

[0073] In one embodiment, in response to a key point being categorized as the tip of the nose and two other key points being categorized as the left shoulder and right shoulder, the midpoints of the line segments corresponding to the key point of the tip of the nose and the left and right shoulders are connected by line segments of the seventh color.

[0074] In one specific embodiment, the pedestrian's joint points include the tip of the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, totaling 13 joint points. A line segment connecting the left ankle and left knee is set to blue, with each pixel of the line segment having a value of (255, 0, 0); a line segment connecting the left knee and left hip is set to green, with each pixel of the line segment having a value of (0, 255, 0); a line segment connecting the left hip and left shoulder is set to red, with each pixel of the line segment having a value of (0, 0, 255); a line segment connecting the left shoulder and left elbow is set to indigo, with each pixel of the line segment having a value of (0, 0, 255). The pixel value of the element is (255, 255, 0); connect the left elbow and left wrist to form a line segment, set this line segment to purple, and set the pixel value of each pixel of the line segment to (255, 0, 255); connect the left shoulder and right shoulder to form a line segment, set this line segment to yellow, and set the pixel value of each pixel of the line segment to (0, 255, 255); take the midpoint of the line segment connecting the left shoulder and right shoulder, and connect it to the tip of the nose to form a line segment, set this line segment to white, and set the pixel value of each pixel of the line segment to (255, 255, 0); (55, 255); Connect the left and right hips with a line segment, set this line segment to white, and the pixel value of each pixel in the line segment is (255, 255, 255); Connect the right ankle and right knee with a line segment, set this line segment to blue, and the pixel value of each pixel in the line segment is (255, 0, 0); Connect the right knee and right hip with a line segment, set this line segment to green, and the pixel value of each pixel in the line segment is (0, 255, 0); Connect the right hip and right shoulder with a line segment, this line segment... The line segment is set to red, with pixel values ​​of (0, 0, 255). A line segment connecting the right shoulder and right elbow is set to indigo, with pixel values ​​of (255, 255, 0). A line segment connecting the right elbow and right wrist is set to purple, with pixel values ​​of (255, 0, 255). This yields the human skeletal structure diagram of the pedestrian in each sample image, carrying different color information. The connecting line segments between joints are fixed, but their colors can be set according to the actual situation. The width of the line segments between key points of the human body can be set according to the actual situation, for example, to 4 pixels.

[0075] This step allows us to obtain the human skeletal structure diagram corresponding to each sample image, and the human skeletal structure diagram of each sample image corresponds one-to-one with the key point information of each sample image and the mask image of the sample image.

[0076] S205: Generate a fused image of the sample images based on the mask image, key point information of the parts, and key point connection structure diagram corresponding to each sample image.

[0077] Specifically, a single-channel mask image is copied and stitched together to obtain a three-channel mask image. For example, the single-channel mask image is copied twice, and the three copies are stitched together to obtain a three-channel mask image. Similarly, single-channel location keypoint information is copied and stitched together to obtain three-channel location keypoint information. For example, single-channel location keypoint information is copied twice, and the three copies are stitched together to obtain three-channel location keypoint information. The three-channel mask image, three-channel location keypoint information, and keypoint connection structure diagram corresponding to the same sample image are fused to generate a fused image of the sample image. The fused image of the sample image is the sum of the mask image, location keypoint information, and human skeletal structure diagram corresponding to the sample image, and includes the connection structure between location keypoints within the target object's region. The fused image is three-channel data.

[0078] Arrange the fused images corresponding to all sample images in the training video stream according to time sequence to generate the fused image sequence corresponding to the training video stream.

[0079] Please see Figure 3 , Figure 3 This is a schematic block diagram of an embodiment of the gait recognition model or teacher model provided by the present invention.

[0080] S206: The teacher model is used to obtain the second predicted category of the target object based on the fused image sequence of the training video stream.

[0081] Specifically, the teacher model includes a feature extraction module and a discrimination module. The feature extraction module extracts features from the fused image sequence to obtain feature vectors of the target object, which contain temporal information. The discrimination module calculates the similarity between the feature vectors and each preset feature vector. The category of the preset vector with the highest similarity is used as the second predicted category of the target object.

[0082] In one specific embodiment, the feature extraction module includes a first feature extraction unit, a second feature extraction unit, a third feature extraction unit, and a 3D convolutional layer, all cascaded together. Each of the first, second, and third feature extraction units includes a convolutional layer and an activation function layer, all cascaded together. The first, second, and third feature extraction units extract features of the target object from the fused image sequence, obtaining feature vectors of the target object in each sample image. The 3D convolutional layer then extracts the temporal information of the target object's feature vectors. The discrimination module includes a first fully connected layer, a second fully connected layer, and a SoftMax layer, all cascaded together. The first and second fully connected layers transform multi-dimensional feature vectors into one-dimensional feature vectors. The SoftMax layer calculates the similarity between the one-dimensional feature vector of the target object and a preset feature vector in the database, thereby determining the probability that the target object and the preset object corresponding to the preset feature vector belong to the same category. The category of the preset object corresponding to the highest probability value is selected as the second predicted category of the target object.

[0083] S207: Iteratively train the teacher model based on the third error value between the second predicted category and the labeled category corresponding to the training video stream.

[0084] Specifically, the teacher model is iteratively trained using a third error value between the second predicted category and the labeled category corresponding to the same training video stream.

[0085] In one optional embodiment, the results of the teacher model are backpropagated, and the weights of the teacher model are corrected based on the third error value between the second predicted category and the labeled category corresponding to the same training video stream, thereby training the teacher model.

[0086] The training video stream is input into the teacher model, which detects the category of the target object in the training video stream. When the third error value between the second predicted category and the labeled category corresponding to the same training video stream is less than a preset threshold (which can be set by the user, such as 1% or 5%), the training of the teacher model is stopped.

[0087] S208: Feature extraction is performed on the fused image sequence of the training video stream using the teacher model to obtain the feature information of the training video stream.

[0088] Specifically, the teacher model extracts features from the fused image sequence of the training video stream to obtain feature information of the target object in the training video stream. This target object feature information serves as supervisory information for training the gait recognition model. Since the teacher network obtains feature information based on the fused image sequence of the training video stream, this feature information contains richer features of the target object. Based on this feature information, the identity information of the target object can be determined more accurately.

[0089] S209: Determine the first predicted category and predicted features of the target object based on the masked image sequence corresponding to the training video stream using a gait recognition model.

[0090] Specifically, the teacher model is a trained model that performs the same task as the gait recognition model. The specific structure of the gait recognition model in this embodiment is the same as that of the teacher model. The difference lies in that the teacher model has 3 channels in the feature extraction layers of the first, second, and third feature extraction units; while the gait recognition model has 1 channel in the feature extraction layers of the first, second, and third feature extraction units.

[0091] The masked image sequence corresponding to the training video stream is input into the gait recognition model. The feature extraction module in the gait recognition model extracts the predicted features of the target object in the training video stream. The discrimination module in the gait recognition model compares the predicted features of the target object with the preset feature information in the database to determine the first predicted category of the target object.

[0092] S210: Iteratively train the gait recognition model based on the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted features.

[0093] Specifically, the final error value is obtained by weighting the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted features. The gait recognition model is then trained iteratively using the final error value. For example, the first error value is calculated based on the cross-entropy loss function; the second error value is calculated based on distillation loss functions such as DKD and Fitnet.

[0094] In one specific embodiment, the final error value Loss is calculated using the following formula 1.

[0095] Loss=α*Kd_loss+β*CE_loss (Formula 1)

[0096] In Formula 1: Loss represents the final error value; Kd_loss represents the first error value; CE_loss represents the second error value; α and β are both weight values.

[0097] In one optional embodiment, the results of the gait recognition model are backpropagated. The weights of the gait recognition model are adjusted using a weighted sum of a first error value between the labeled category and the first predicted category corresponding to the same training video stream, and a second error value between the feature information and the predicted features. This process trains the gait recognition model. The gait recognition model can then determine the category of the target object corresponding to the training video stream based on the masked image sequence, ensuring that the category of the target object obtained by the gait recognition model based on single-channel data is consistent with the category of the target object obtained by the teacher model based on three-channel data.

[0098] The training video stream is input into the gait recognition model, which detects the category of the target object in the training video stream. When the weighted sum of the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted feature is less than a preset threshold, the training of the gait recognition model is stopped. The preset threshold can be set by the user, such as 1% or 5%.

[0099] In this embodiment, the teacher model generates feature information of the training video stream based on the fused image sequence of the training video stream. This feature information is used as supervision information. Then, the gait recognition model determines the first predicted category and predicted features of the target object based on the mask image sequence of the training video stream. The gait recognition model is trained by the error value between the predicted features and the supervision information and the error value between the first predicted category and the labeled category. This allows the gait recognition model to obtain the same or similar target categories and target features based on the mask image sequence of single-channel data or the fused image sequence, thereby improving the recognition accuracy of the gait recognition model and reducing the amount of computation, thus shortening the recognition time.

[0100] Please see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the gait recognition method provided by the present invention. This embodiment provides a gait recognition method, which includes the following steps.

[0101] S31: Get the video stream.

[0102] Specifically, the video stream comprises multiple video frames containing the same target. Images containing pedestrians are captured using image acquisition devices such as surveillance cameras and dashcams to obtain the video stream containing the same target. In other words, the video stream consists of multiple video frames of the same target taken from different shooting angles. In this embodiment, the target is a pedestrian.

[0103] S32: Segment each video frame to obtain a mask image sequence of the video stream.

[0104] Specifically, background subtraction or a trained segmentation model is used to segment each video frame in the video stream to obtain the pedestrian contour map corresponding to each video frame.

[0105] In one embodiment, target detection is performed on the video frame, and the video frame is then binarized. Specifically, the pixels at the target location in the video frame are assigned a value of 1, and the pixels at the background location are assigned a value of 0, resulting in a mask image corresponding to the video frame. In this embodiment, the mask image is specifically a pedestrian outline image. In this embodiment, the mask image is single-channel data.

[0106] This step allows us to obtain the mask image corresponding to each video frame, and then arrange the mask images corresponding to all video frames in the video stream according to time sequence to generate a mask image sequence for the video stream. The mask image sequence is single-channel data.

[0107] S33: Use a gait recognition model to determine the target's identity category based on the masked image sequence corresponding to the video stream.

[0108] Specifically, the sequence of masked images corresponding to the video stream is input into the gait recognition model trained through the above embodiments. The feature extraction module in the gait recognition model extracts features from each masked image in the sequence and extracts the temporal information of each masked image, thereby obtaining a multi-dimensional feature vector of the target in the video stream. The discrimination module in the gait recognition model performs dimensionality reduction processing on the multi-dimensional feature vector obtained by the feature extraction module to obtain a one-dimensional feature vector. Then, the one-dimensional feature vector is compared with a preset feature vector in the database, and the preset identity category corresponding to the preset feature vector with the highest similarity to the target's feature vector is output as the identity category of the target in the video stream.

[0109] This embodiment provides a gait recognition method. The gait recognition method segments each video frame in the acquired video stream to obtain a mask image sequence of the video stream. Based on the mask image sequence corresponding to the video stream, a gait recognition model is used to determine the identity category of the target. The gait recognition method can determine the identity category of the target corresponding to the video stream through a single-channel mask image sequence, which improves the accuracy of target identity category recognition, avoids complex data processing, and reduces the time consumption of the target identity category recognition process.

[0110] See Figure 5 , Figure 5 This is a schematic diagram of the framework of an embodiment of the gait recognition model training device provided by the present invention. This embodiment provides a gait recognition model training device 60, which includes an acquisition module 61, a supervision module 62, a processing module 63, and a training module 64.

[0111] The acquisition module 61 is used to acquire the training video stream, which contains multiple frames of sample images with the same target object. The training video stream is associated with a corresponding mask image sequence and a fused image sequence, and the fused image sequence is multi-channel data, while the mask image sequence is single-channel data. The training video stream has the labeled category of the target object.

[0112] The supervision module 62 is used to extract features from the fused image sequence of the training video stream using the teacher model, thereby obtaining the feature information of the training video stream.

[0113] The processing module 63 is used to determine the first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream through the gait recognition model; the teacher model is a model that has been trained and performs the same task as the gait recognition model.

[0114] The training module 64 is used to iteratively train the gait recognition model based on the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted features.

[0115] The gait recognition model training device provided in this embodiment generates feature information of the training video stream based on the fused image sequence of the training video stream by the teacher model. This feature information is used as supervision information. Then, the gait recognition model determines the first predicted category and predicted features of the target object based on the mask image sequence of the training video stream. The gait recognition model is trained by the error value between the predicted features and the supervision information and the error value between the first predicted category and the labeled category. This allows the gait recognition model to obtain the same or similar target categories and target features based on the mask image sequence of single-channel data or the fused image sequence, thereby improving the recognition accuracy of the gait recognition model and reducing the amount of computation, thus shortening the recognition process time.

[0116] See Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the gait recognition device provided by the present invention. This embodiment provides a gait recognition device 70, which includes an information acquisition module 71, a preprocessing module 72, and a determination module 73.

[0117] The information acquisition module 71 is used to acquire video streams, which include multiple video frames containing the same target.

[0118] The preprocessing module 72 is used to segment each video frame to obtain a mask image sequence of the video stream;

[0119] The determination module 73 is used to determine the identity category of the target based on the mask image sequence corresponding to the video stream using a gait recognition model; wherein, the gait recognition model is trained by the gait recognition model training method in the above embodiment.

[0120] This embodiment provides a gait recognition device. The gait recognition device segments each video frame in the acquired video stream to obtain a mask image sequence of the video stream. Based on the mask image sequence corresponding to the video stream, a gait recognition model is used to determine the identity category of the target. The gait recognition method can determine the identity category of the target corresponding to the video stream through a single-channel mask image sequence, which improves the accuracy of target identity category recognition, avoids complex data processing, and reduces the time consumption of the target identity category recognition process.

[0121] Please see Figure 7 , Figure 7 This is a schematic diagram of a terminal embodiment provided in this application. The terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described gait recognition model training methods or gait recognition method embodiments. In a specific implementation scenario, the terminal 80 may include, but is not limited to, a microcomputer or a server. In addition, the terminal 80 may also include mobile devices such as laptops and tablets, which are not limited here.

[0122] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described gait recognition model training methods or gait recognition method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0123] Please see Figure 8 , Figure 8This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described gait recognition model training methods or gait recognition method embodiments.

[0124] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0125] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0130] The above are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for training a gait recognition model, characterized in that, include: A training video stream is obtained, which contains multiple frames of sample images with the same target object. The training video stream is associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has the labeled category of the target object. The feature information of the training video stream is obtained by extracting features from the fused image sequence of the training video stream using a teacher model. The gait recognition model determines the first predicted category and predicted features of the target object based on the masked image sequence corresponding to the training video stream; the teacher model is a model that has been trained and performs the same task as the gait recognition model. Based on the first error value between the labeled category and the first predicted category corresponding to the same training video stream and the second error value between the feature information and the predicted feature, the gait recognition model is iteratively trained. The acquisition of the training video stream also includes: Each of the sample images is segmented to obtain a mask image of the sample image; Key point detection is performed on each of the sample images to obtain key point information of the corresponding parts of the sample images; Based on the key point information of the sample image, a key point connection structure diagram corresponding to the sample image is generated. Based on the mask image, key point information of the part, and key point connection structure diagram corresponding to each sample image, a fused image of the sample images is generated; the fused image is multi-channel data. Arrange the fused images corresponding to all the sample images in the training video stream in temporal order to generate the fused image sequence corresponding to the training video stream; The training method for the teacher model includes: The teacher model is used to obtain the second predicted category of the target object based on the fused image sequence of the training video stream; The teacher model is iteratively trained based on the third error value between the second predicted category and the labeled category corresponding to the training video stream.

2. The gait recognition model training method according to claim 1, characterized in that, The acquisition of the training video stream includes: The mask images corresponding to all the sample images in the training video stream are arranged in time sequence to generate the mask image sequence corresponding to the training video stream.

3. The gait recognition model training method according to claim 2, characterized in that, The fused image is three-channel data; The step of generating a fused image of the sample images based on the mask image corresponding to each sample image, the key point information of the part, and the key point connection structure diagram includes: The single-channel mask image is copied and stitched together to obtain a three-channel mask image; The key point information of the part in the single channel is copied and spliced ​​to obtain the key point information of the part in the three channel; The three-channel mask image, the key point information of the three-channel region, and the key point connection structure diagram corresponding to the same sample image are fused to generate the fused image of the sample image.

4. The gait recognition model training method according to claim 2, characterized in that, The key point information of the part includes the key points of the part, as well as the category and location of the key points of the part; The step of generating a key point connection structure diagram corresponding to the sample image based on the key point information of the sample image includes: Based on the category and location of the key points in the region, the key points in the region are connected by line segments of different types to obtain the key point connection structure diagram corresponding to the sample image.

5. The gait recognition model training method according to claim 4, characterized in that, The categories of key points in the aforementioned locations include the tip of the nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and / or right ankle; The step of connecting the key points of the region with different types of line segments based on the category and location of the key points of the region to obtain the key point connection structure diagram corresponding to the sample image includes: In response to one of the key points of the body part being classified as the left ankle and the other key point of the body part being classified as the left knee, and / or one key point of the body part being classified as the right ankle and the other key point of the body part being classified as the right knee, the two key points of the body part are connected by the line segment of the first color; And / or; in response to one of the key points of the body part being classified as the left hip and the other key point of the body part being classified as the left knee, and / or one key point of the body part being classified as the right hip and the other key point of the body part being classified as the right knee, then the two key points of the body part are connected by the line segment of the second color; And / or; in response to one of the key points of the body being classified as the left hip and the other key point of the body being classified as the left shoulder, and / or one key point of the body being classified as the right hip and the other key point of the body being classified as the right shoulder, then the two key points of the body are connected by the line segment of the third color; And / or; in response to one of the location key points being classified as the left elbow and the other location key point being classified as the left shoulder, and / or one of the location key points being classified as the right elbow and the other location key point being classified as the right shoulder, then the two location key points are connected by the line segment of the fourth color; And / or; in response to one of the location key points being classified as the left elbow and the other location key point being classified as the left wrist, and / or one of the location key points being classified as the right elbow and the other location key point being classified as the right wrist, then the two location key points are connected by the line segment of the fifth color; And / or; in response to one of the key points of the body part being classified as the right shoulder and the other key point of the body part being classified as the left shoulder, and / or one key point of the body part being classified as the right hip and the other key point of the body part being classified as the left hip, then the two key points of the body part are connected by the line segment of the sixth color; And / or; in response to one of the key points of the body part being classified as the tip of the nose and the other two key points of the body part being classified as the left shoulder and the right shoulder respectively, the midpoint of the key point of the body part corresponding to the tip of the nose and the midpoint of the line segment corresponding to the left shoulder and the right shoulder are connected by the line segment of the seventh color.

6. The gait recognition model training method according to claim 1, characterized in that, The teacher model includes a feature extraction module and a discrimination module. The step of obtaining the second predicted category of the target object using the teacher model based on the fused image sequence of the training video stream includes: The feature extraction module is used to extract features from the fused image sequence to obtain the feature vector of the target object, and the feature vector contains temporal information; The discrimination module is used to calculate the similarity between the feature vector and each preset feature vector; The category of the preset feature vector corresponding to the highest similarity is used as the second predicted category of the target object.

7. A gait recognition method, characterized in that, The gait recognition method includes: Acquire a video stream, the video stream comprising multiple video frames containing the same target; Each of the video frames is segmented to obtain a mask image sequence of the video stream; The identity category of the target is determined by using a gait recognition model based on the mask image sequence corresponding to the video stream; wherein the gait recognition model is trained by the gait recognition model training method according to any one of claims 1 to 6.

8. A gait recognition model training device, characterized in that, The gait recognition model training device includes: An acquisition module is used to acquire a training video stream, which contains multiple frames of sample images with the same target object. The training video stream is associated with a corresponding mask image sequence and a fused image sequence, wherein the fused image sequence is multi-channel data and the mask image sequence is single-channel data; the training video stream has a labeled category of the target object; the module is also used to segment each of the sample images to obtain a mask image of the sample image; perform keypoint detection on each of the sample images to obtain keypoint information of the corresponding parts of the sample image; generate a keypoint connection structure diagram corresponding to the sample image based on the keypoint information of the parts of the sample image; generate a fused image of the sample image according to the mask image, the keypoint information of the parts, and the keypoint connection structure diagram corresponding to each of the sample images; the fused image is multi-channel data; and arrange the fused images corresponding to all the sample images in the training video stream in temporal order to generate the fused image sequence corresponding to the training video stream. The supervision module is used to extract features from the fused image sequence of the training video stream using a teacher model to obtain feature information of the training video stream; it is also used to use the teacher model to obtain a second predicted category of the target object based on the fused image sequence of the training video stream; and to iteratively train the teacher model based on a third error value between the second predicted category and the labeled category corresponding to the training video stream. The processing module is used to determine the first predicted category and predicted features of the target object based on the mask image sequence corresponding to the training video stream using a gait recognition model; the teacher model is a model that has been trained and performs the same task as the gait recognition model. The training module is used to iteratively train the gait recognition model based on a first error value between the labeled category and the first predicted category corresponding to the same training video stream and a second error value between the feature information and the predicted feature.

9. A gait recognition device, characterized in that, The gait recognition device includes: The information acquisition module is used to acquire video streams, which include multiple video frames containing the same target. The preprocessing module is used to segment each of the video frames to obtain a mask image sequence of the video stream; The determination module is used to determine the identity category of the target based on the mask image sequence corresponding to the video stream using a gait recognition model; wherein the gait recognition model is trained by the gait recognition model training method according to any one of claims 1 to 6.

10. A terminal, characterized in that, The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor. The processor is used to execute program data to implement the steps of the gait recognition model training method as described in any one of claims 1 to 6 or the gait recognition method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the gait recognition model training method as described in any one of claims 1 to 6 or the gait recognition method as described in claim 7.

Images