Pedestrian recognition method and device, and electronic device

Abstract

This application discloses a pedestrian recognition method, belonging to the field of computer vision technology, which helps to improve the accuracy of pedestrian recognition. The method includes: acquiring the contour information of pedestrians included in a target image; generating a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image, excluding the human head, is masked; extracting head features from the mask image, and extracting human morphological features from the mask image, and fusing the features obtained from the head feature extraction and the human morphological feature extraction respectively; and recognizing the pedestrian based on the fused features. This method reduces interference from invalid information introduced by the background and pedestrian clothing by masking the target image, and improves the accuracy of pedestrian recognition by extracting the head features and human morphological features of the pedestrian from the mask image.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to pedestrian recognition methods, devices, electronic devices, and computer-readable storage media. Background Technology

[0002] Pedestrian re-identification is a technique that uses computer vision to determine whether a specific pedestrian exists in a video image. For example, in non-cooperative attendance scenarios, it involves identifying and detecting target pedestrians in video images.

[0003] In recent years, deep learning technology has developed rapidly and has been widely applied in the field of image classification. For example, many deep learning-based pedestrian recognition schemes have emerged. Existing pedestrian recognition schemes mainly include the following steps: inputting the detected pedestrian image into a deep learning model to extract feature maps; calculating the similarity between the extracted feature maps and the features in a pedestrian database; and determining the pedestrian recognition result based on the calculated similarity. However, existing pedestrian recognition methods have low accuracy when dealing with pedestrians who have changed clothes or other unusual situations.

[0004] It is evident that existing pedestrian recognition methods still require improvement. Summary of the Invention

[0005] This application provides a pedestrian recognition method and apparatus, which helps to improve the accuracy of pedestrian recognition.

[0006] In a first aspect, embodiments of this application provide a pedestrian identification method, including:

[0007] Obtain the outline information of pedestrians included in the target image;

[0008] A mask image is generated based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image other than the human head is masked.

[0009] The mask image is processed by a first feature processing module to extract head features, and by a second feature processing module to extract human body morphology features. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features.

[0010] The pedestrian is identified based on the fused features.

[0011] Secondly, embodiments of this application provide a pedestrian identification device, including:

[0012] The pedestrian contour information acquisition unit is used to acquire the contour information of pedestrians included in the target image;

[0013] A mask image generation unit is used to generate a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image other than the human head is masked.

[0014] The feature extraction unit is used to extract head features from the mask image using a first feature processing module, and to extract human body morphology features from the mask image using a second feature processing module. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features.

[0015] A pedestrian recognition unit is used to identify the pedestrian based on the fused features.

[0016] Thirdly, embodiments of this application also disclose an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the pedestrian recognition method described in embodiments of this application.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, constitutes the steps of the pedestrian identification method disclosed in embodiments of this application.

[0018] The pedestrian recognition method disclosed in this application involves acquiring the contour information of pedestrians included in a target image; generating a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image, excluding the human head, is masked; extracting head features from the mask image using a first feature processing module, and extracting human morphological features from the mask image using a second feature processing module; then fusing the features obtained from the head feature extraction and the human morphological feature extraction to obtain fused features; and recognizing the pedestrian based on the fused features, which helps improve the accuracy of pedestrian recognition.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Figure 1 This is a flowchart of the pedestrian recognition method according to Embodiment 1 of this application;

[0022] Figure 2 This is a schematic diagram of the human-shaped segmentation model structure used in Embodiment 1 of this application;

[0023] Figure 3 This is a schematic diagram of the mask image described in Embodiment 1 of this application;

[0024] Figure 4 This is another flowchart of the pedestrian identification method according to Embodiment 1 of this application;

[0025] Figure 5 This is a schematic diagram of the pedestrian recognition model structure used in the pedestrian recognition stage of Embodiment 1 of this application;

[0026] Figure 6 This is a schematic diagram of the pedestrian recognition model structure used in the model training phase of Embodiment 1 of this application;

[0027] Figure 7 This is one of the schematic diagrams of the pedestrian recognition device structure in Embodiment 2 of this application;

[0028] Figure 8 This is the second schematic diagram of the pedestrian recognition device structure in Embodiment 2 of this application;

[0029] Figure 9 A block diagram schematically illustrates an electronic device for performing the method according to this application; and

[0030] Figure 10 A storage unit for holding or carrying program code implementing the method according to this application is illustrated schematically. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Example 1

[0033] This application discloses a pedestrian identification method, such as... Figure 1 As shown, the method includes steps 110 to 140.

[0034] Step 110: Obtain the outline information of pedestrians included in the target image.

[0035] In this embodiment, the target image refers to an image that includes one pedestrian. For example, the target image can be an image containing only one pedestrian, extracted from the surveillance video image based on pedestrian location boxes obtained by pedestrian detection of the video image. The target image is a color image that includes both the pedestrian and the background.

[0036] The target image is an image containing only one pedestrian after preprocessing the video image. When there are two or more pedestrians and they overlap, a complete pedestrian image is extracted as the target image.

[0037] For a received target image, the first step is to obtain the outline information of the pedestrians included in the image. The outline of the human body can not only reflect overall morphological information such as height, weight, and build, but also morphological information of local body parts such as body proportions. Therefore, it can be used as a reference for pedestrian recognition.

[0038] In some embodiments of this application, principal component analysis can be used to perform foreground and background analysis on the target image, with pedestrians as the foreground, to perform foreground segmentation on the target image, so as to obtain the location information of the foreground region in the target image, that is, the outline information of the pedestrians included in the target image.

[0039] In some embodiments of this application, the contour information of pedestrians included in the target image can be represented by a binary feature map. For example, for a target image of size H*W, the contour information of pedestrians included in the target image can be represented by an H*W feature map, in which the feature value of the image area covered by the pedestrian is set to 0, and the feature value of the other image areas is set to 255.

[0040] In other embodiments of this application, the contour information of pedestrians included in the target image can also be obtained through deep learning methods. For example, obtaining the contour information of pedestrians included in the target image includes: performing human segmentation on the target image using a pre-trained human segmentation model to obtain the contour information of pedestrians included in the target image. The human segmentation model can be an existing human segmentation model in the prior art, or a corresponding model structure can be designed and trained autonomously based on the specific application scenario of this application.

[0041] In some embodiments of this application, the human figure segmentation model is trained using the following method: for each sample image, the human figure segmentation model extracts edge contour features of the sample image through a built-in top-level convolutional layer, and extracts deep semantic features of the sample image through a built-in bottom-level convolutional layer; the edge contour features and the deep semantic features are fused, and feature mapping is performed based on the fused features to obtain a contour information feature map; the human figure segmentation model is iteratively trained with the goal of optimizing the contour information feature map.

[0042] FCOS (Fully Convolutional One-Stage Object Detection) is an existing object detection model. The FCOS model eliminates anchors for detection, making it an anchor-free object detection model. The core idea of ​​FCOS is to predict the object category and bounding box for each point in the input image.

[0043] In some embodiments of this application, the FCOS model is constructed using a similar architecture to FPN (Feature Pyramid Network). By extracting features from the input image at different scales through different convolutional layers and forming a pyramid shape, it is beneficial to fully utilize the semantic information at different scales within the feature pyramid. The structure of FPN can be summarized as: feature extraction, upsampling, feature fusion, and multi-scale feature output. The input to FPN is an image of arbitrary size, and the output is a feature map at various scales. The entire network structure of FPN is divided into two parts: bottom-up and top-down. Bottom-up is the feature extraction process. Top-down upsampling involves upsampling the deepest features layer by layer to the resolution corresponding to the bottom-up output. This top-down upsampling is then fused with the bottom-up features to output a feature map, with corresponding positions being added together. The output layer of the FCOS model makes predictions based on the fused features.

[0044] The FCOS model has three outputs, and the function of each output is described below.

[0045] The first output of the FCOS model is the classification branch, which can be represented as an H*W*C feature vector, where H*W represents the feature size and C represents the number of classes. The position (x, y) on the feature can be converted into the position in the input image using a preset formula based on a corresponding scaling factor. This establishes a relationship between the positions of points on the output feature map and the positions of points in the input image, facilitating the calculation of the classification and regression targets for each point on the feature map. Furthermore, taking C=2 classes as an example, where the two classes represent pedestrians and background respectively, the output of the classification branch can determine whether each position in the input image is a pedestrian or background.

[0046] The second output of the FCOS model is used to calculate the distance between each point and the target center point, represented by an H*W*1 feature map.

[0047] The third output of the FCOS model is the regression branch, which is used to regress localization parameters.

[0048] In some embodiments of this application, the following are employed: Figure 2 The FCOS model shown includes: a convolutional neural network 210, a feature pyramid network 220, an output layer 230, an attention mechanism network 240, and a feature fusion network 250. The attention mechanism network 240 fuses the top-level features output by the convolutional neural network 210 using an attention mechanism to obtain edge contour features of the input image. These edge contour features better express contour information in the image (such as the contour information of pedestrians). The feature pyramid network 220 further extracts deep semantic features from the hidden layer features output by the convolutional neural network 210 through multi-scale feature extraction. These deep semantic features more accurately express the detailed information of the image (such as color and texture information). The feature fusion network 250 fuses the edge contour features output by the attention mechanism network 240 and the deep semantic features output by the feature pyramid network 220 to obtain fused features at various scales, which are then input to the output layer 230. The output layer 230 performs feature mapping based on the fused features to predict the contour information feature map of the input image.

[0049] In some embodiments of this application, for each input image used as a training sample, a true value for pedestrian contour information is pre-set. During the training phase of the human segmentation model, the loss of the human segmentation model can be calculated by calculating the error between the true value of pedestrian contour information in each sample image and the predicted contour information feature map, and the human segmentation model is iteratively trained with the goal of minimizing the loss.

[0050] Based on the prediction results and true contour information of the human figure segmentation model, the specific implementation method for optimizing and training the human figure segmentation model is described in the prior art optimization process, which will not be repeated in the embodiments of this application.

[0051] Step 120: Generate a mask image based on the contour information and the target image.

[0052] The masked image includes the pedestrian's image within the target image, and the pedestrian's image area in the masked image, excluding the head, is masked. Figure 3 Taking the target image on the left as an example, the generated mask image is as follows: Figure 3 As shown in the image on the right.

[0053] That is, the masked image includes: the original head image of the pedestrian in the target image, and the torso morphological image of the pedestrian in the target image, wherein the morphological image is a masked image.

[0054] In some embodiments of this application, the mask image is an image in which the background has a first color value, the torso of the pedestrian has a second color value, and the head of the pedestrian is a color image.

[0055] In some embodiments of this application, generating a mask image based on the contour information and the target image includes: determining a human-shaped image region in the target image containing a pedestrian based on the contour information; determining a human head sub-region within the human-shaped image region; generating a mask image based on the image content of the human head sub-region in the target image and the morphological information of the human-shaped image region; wherein the morphological information of the human-shaped image region in the mask image is consistent with the morphological information of the human-shaped image region in the target image; and, the image pixel values ​​outside the human-shaped image region in the mask image are first color values, the image pixel values ​​within the human-shaped image region but outside the human head sub-region in the mask image are second color values, and the image content of the human head sub-region within the human-shaped image region in the mask image is consistent with the image content of the human head sub-region in the target image. The first color value and the second color value are different values.

[0056] As mentioned above, the contour information represents the image area covered by the pedestrian in the target image. Therefore, based on the contour information, the image area where the pedestrian's image is located in the target image can be further determined, i.e., the pedestrian's image area, referred to as the "human image area" in this embodiment. Further, based on the height ratio of the human head and torso, the image area covered by the human head in the human image area (hereinafter referred to as the "human head sub-region") and the image area covered by the human torso (i.e., body parts other than the human head in the human image area, such as the torso, arms, legs, hands, and feet) (hereinafter referred to as the "human torso sub-region") can be determined. In some embodiments of this application, the top 1 / 4 of the human image area can be taken as the human head sub-region, and the image area outside the human head sub-region in the human image area can be taken as the human torso sub-region.

[0057] Next, a mask image is generated based on the image content of the human head sub-region in the target image and the morphological information of the human-shaped image region. For example, a mask image is generated as follows: Figure 3 The image shown is illustrated. In this embodiment, the generated mask image includes two main image regions: a human figure image region and a background region. The human figure image region is further divided into two sub-regions: a human head sub-region and a human torso sub-region. In this embodiment, a mask image of the same scale as the target image can be generated. Then, a human figure image region identical to the human figure image region in the target image is generated within the mask image. The morphological information of the human figure image region reflects the morphological information of the pedestrian's outline and characterizes the pedestrian's body features. Furthermore, by setting the image pixel values ​​of the background region and the human torso sub-region to two different color values, the morphological features of the pedestrian are preserved. The image content of the human head sub-region in the mask image is set to match the human head image in the target image to preserve and highlight the pedestrian's head features.

[0058] In practice, various methods can be used to generate a mask image based on the image content of the human head sub-region in the target image and the morphological information of the human figure image region. The following two examples illustrate how to obtain a mask image based on the morphological information of the human figure image region and the image content of the human head sub-region in the target image.

[0059] In some embodiments of this application, generating a mask image based on the image content of the human head sub-region in the target image and the morphological information of the human-shaped image region includes: setting the image pixel values ​​outside the human-shaped image region in the target image to a first color value, and setting the image pixel values ​​within the human-shaped image region but outside the human head sub-region in the target image to a second color value, thereby obtaining a mask image. The first color value and the second color value are different pixel values. For example, in some embodiments of this application, the first color value is 0 (i.e., black), and the second color value is 255 (i.e., white), resulting in a mask image as shown below. Figure 3 As shown.

[0060] In some other embodiments of this application, generating a mask image based on the image content of the human head sub-region in the target image and the morphological information of the human figure image region includes: creating a mask image, the mask image including a mask region with the same morphological information as the human figure image region, wherein the image pixel value within the mask region is a second color value; initializing the image pixel values ​​outside the mask region in the mask image to a first color value; and copying the image content of the human head sub-region in the target image to the corresponding pixel position in the mask image. For example, firstly, a black and white image with the same scale as the target image is created, wherein the image region corresponding to the human figure image region in the target image is set to white, and other regions are set to black. Then, the pixel values ​​of all pixels in the human head sub-region of the target image are assigned one-to-one to the pixel positions in the corresponding image positions in the mask image. This results in a mask image where the human head is identical to the head in the target image, the human body region is masked as white, and the background is masked as black.

[0061] The above are merely two examples of generating masked images. Those skilled in the art can also generate masked images using other methods, based on the morphological information of a determined human-shaped image region and the human head image in the target image. These will not be listed in detail in the embodiments of this application. Furthermore, those skilled in the art should understand that the masked image generation schemes based on the idea disclosed in this application—generating masked images based on the morphological information of a determined human-shaped image region and the human head image in the target image—should fall within the protection scope of this application.

[0062] Step 130: The first feature processing module is used to extract head features from the mask image, and the second feature processing module is used to extract human body morphology features from the mask image. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features.

[0063] The first feature processing module and the second feature processing module are two neural network modules with different structures.

[0064] Subsequently, features for identification are further extracted from the generated mask image using a pre-trained neural network model.

[0065] In some embodiments of this application, two feature extraction networks with different structures (such as a first feature processing module and a second feature processing module) are used to extract features from the mask image to obtain two sets of different features extracted from the mask image based on different dimensions. These two sets of features are then fused for pedestrian recognition.

[0066] In some embodiments of this application, a pedestrian recognition model is used to extract and fuse features from the masked image.

[0067] like Figure 4 As shown, in some embodiments of this application, before the first feature processing module in the pre-trained pedestrian recognition model extracts head features from the mask image to obtain the first feature of the target image, and before the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the target image, the method further includes: step 100.

[0068] Step 100: Train the pedestrian recognition model based on several masked images with category labels.

[0069] like Figure 5 As shown, the pedestrian recognition model may include: a first feature processing module 510, a second feature processing module 520, and a feature fusion module 530.

[0070] In some embodiments of this application, the first feature processing module extracts head features from the mask image, and the second feature processing module extracts human morphological features from the mask image. Then, the features obtained from the head feature extraction and the human morphological feature extraction are fused to obtain fused features. This includes: extracting head features from the mask image using the first feature processing module in a pre-trained pedestrian recognition model to obtain a first feature of the target image; and extracting human morphological features from the mask image using the second feature processing module in the pedestrian recognition model to obtain a second feature of the target image. The first feature processing module and the second feature processing module are built based on different neural network structures. The first feature and the second feature are then fused to obtain fused features.

[0071] In some embodiments of this application, the first feature processing module (i.e., the first feature processing module 510 of the pedestrian recognition model) is a convolutional neural network model suitable for extracting facial features. For example, the first feature processing module can be built using the structure of MobileFaceNet (a facial feature extraction network) in the prior art. MobileFaceNet is a convolutional neural network composed of convolutional layers and separable convolutions. Since the image of the human head sub-region in the mask image is unique relative to the overall human image and represents a strong correlation, in the structural design of the first feature processing module, the outputs of multiple separable convolutions of, for example, the MobileFaceNet network can be further connected to an attention mechanism network, so that the first feature processing module can learn the strongly correlated features of the pedestrian's head, thereby outputting a first feature strongly correlated with the pedestrian.

[0072] In some embodiments of this application, the second feature processing module (i.e., the second feature processing module 520 of the pedestrian recognition model) is a network suitable for extracting contour features. For example, the second feature processing module can be built using a VIT network structure. The VIT (Vision Transformer) network structure extracts features by dividing the input image into blocks and generating a sequence of image blocks, which can improve the ability of the extracted second features to express the pedestrian contour.

[0073] In some embodiments of this application, a feature fusion module 530 is provided after the first feature processing module 510 and the second feature processing module 520 of the pedestrian recognition model. The feature fusion module 530 is used to fuse the first feature output by the first feature processing module and the second feature output by the second feature processing module, and output the fused feature. For example, the feature fusion module 530 can use a 1*1 convolution kernel to concatenate the features output by the first feature processing module and the features output by the second feature processing module to complete the model fusion and obtain the fused feature.

[0074] To enable readers to gain a deeper understanding of the specific scheme for extracting fusion features from masked images, the training process of the pedestrian recognition model used in the embodiments of this application will be further described in detail below.

[0075] In some embodiments of this application, the mask image used to train the pedestrian recognition model includes a human figure image, wherein the human head image of the human figure image is a color image, and the image pixel values ​​of the human figure image in the mask image other than the human head image area are second color values, while the image pixel values ​​of the image area outside the human figure image in the mask image are first color values.

[0076] The method for generating the mask image used to train the pedestrian recognition model is described in the specific implementation of generating the mask image based on the target image in the aforementioned steps, and will not be repeated here.

[0077] In the embodiments of this application, the category label of the mask image used to train the pedestrian recognition model is used to represent the true value of the category matched by the mask image, such as the identifier of the pedestrian to which the pedestrian image that generated the mask image belongs.

[0078] In some embodiments of this application, training the pedestrian recognition model based on several masked images with category labels includes: for each masked image, performing the following feature extraction and classification operations: extracting head features from the masked image using a first feature processing module in the pedestrian recognition model to obtain a first feature of the masked image; and extracting human morphological features from the masked image using a second feature processing module in the pedestrian recognition model to obtain a second feature of the masked image; fusing the first feature and the second feature to obtain a fused feature of the masked image; performing classification mapping on the fused feature to obtain a predicted classification result value of the masked image; calculating the classification loss of the pedestrian recognition model based on the predicted classification result value of each masked image and the category label of the corresponding masked image, and training the pedestrian recognition model with the goal of optimizing the classification loss.

[0079] For details on the specific implementation of extracting head features from the mask image using the first feature processing module in the pedestrian recognition model to obtain the first feature of the mask image, and extracting human morphological features from the mask image using the second feature processing module in the pedestrian recognition model to obtain the second feature of the mask image, please refer to the relevant description of the pedestrian recognition stage above, which will not be repeated here.

[0080] For a detailed implementation of fusing the first feature and the second feature to obtain the fused features of the mask image, please refer to the relevant description of the pedestrian recognition stage above, which will not be repeated here.

[0081] During the pedestrian recognition model training phase, the model further performs classification mapping on the fused features to determine the predicted classification result of the input mask image. The training process of the pedestrian recognition model involves continuously optimizing the parameters of the first and second feature processing modules, thereby making the predicted classification result of the mask image used as a training sample gradually approach the category label (i.e., the true category value).

[0082] During the training phase of a pedestrian recognition model, the classification loss can be calculated using the Triplet Loss function as shown in the following formula: L = max(d(a,p) - d(a,n) + margin, 0); where d(a,p) represents the distance between the mask image a (as an anchor example) and the mask image p (which belongs to the same category as the positive example), d(a,n) represents the distance between the mask image a (as an anchor example) and the mask image n (which belongs to a different category as the negative example), and margin represents the boundary value. The distance between the mask images can be represented by the similarity distance between the fused features of the mask images.

[0083] In other embodiments of this application, during the pedestrian recognition model training phase, such as Figure 6 As shown, the pedestrian recognition model includes a first feature processing module 510, a second feature processing module 520, a spatial rotation module 540, a first classification mapping module 550, and a second classification mapping module 560. Specifically, the first classification mapping module 550 performs classification mapping on the first feature output by the first feature processing module 510 and outputs a predicted first classification result for the input mask image; the spatial rotation module 540 performs a spatial transformation of the second feature output by the second feature processing module 520 at a specified angle to obtain a third feature of the input mask image; and the second classification mapping module 560 performs classification mapping on the third feature output by the spatial rotation module 540 and outputs a predicted second classification result for the input mask image.

[0084] In some embodiments of this application, training the pedestrian recognition model based on several masked images with category labels includes: for each masked image, performing the following feature extraction and classification operations: extracting head features from the masked image using a first feature processing module in the pedestrian recognition model to obtain a first feature of the masked image; and extracting human morphological features from the masked image using a second feature processing module in the pedestrian recognition model to obtain a second feature of the masked image; performing a spatial rotation transformation of the second feature at a random angle to obtain a third feature of the masked image; performing classification mapping on the first feature to obtain a first classification result prediction value of the masked image; and performing classification mapping on the third feature to obtain a second classification result prediction value of the masked image; calculating the classification loss of the pedestrian recognition model based on the first classification result prediction value, the second classification result prediction value, and the category label of the corresponding masked image for each masked image, and training the pedestrian recognition model with the goal of optimizing the classification loss.

[0085] For details on the specific implementation of extracting head features from the mask image using the first feature processing module in the pedestrian recognition model to obtain the first feature of the mask image, and extracting human morphological features from the mask image using the second feature processing module in the pedestrian recognition model to obtain the second feature of the mask image, please refer to the relevant description of the pedestrian recognition stage above, which will not be repeated here.

[0086] In some embodiments of this application, an image of a pedestrian is processed by inputting a mask image generated from the acquired image of the pedestrian into a second feature processing module for feature extraction. After obtaining the second feature, the second feature is further multiplied by a rotation coefficient to perform a spatial transformation, thereby obtaining morphological features, i.e., the third feature, that carry the pedestrian from different angles. In some embodiments of this application, the rotation coefficient can be represented by the cosine value of a random angle, which can be set to any angle from the set of 30 degrees, 60 degrees, 90 degrees, and 180 degrees.

[0087] During the pedestrian registration phase, to improve the recognition accuracy of pedestrian images captured by cameras positioned at different angles, images of the pedestrian's front and side views from different angles are typically collected to extract multiple sets of template features from different angles. To further improve subsequent recognition accuracy, during the pedestrian recognition model training phase, a spatial rotation module is added after the second feature processing module. A preset rotation coefficient (such as the cosine of a random angle from a preset set of rotation angles) is randomly selected to perform spatial transformation on the extracted second features. The fused and transformed features are then used for classification matching against a mask image used as training samples. This completes the training of the pedestrian recognition model, enabling it to have a stronger ability to adapt to different pedestrian image shooting angles.

[0088] During the pedestrian recognition model training phase, the first feature processing module 510 and the first classification mapping module 550 constitute one network branch, while the second feature processing module 520, the spatial rotation module 540, and the second classification mapping module 560 constitute another network branch. The two network branches learn different mapping relationships from different feature perspectives based on the same training samples, thereby obtaining a comprehensive and accurate feature representation of the mask image.

[0089] In some embodiments of this application, different loss functions can be used to optimize the two network branches respectively.

[0090] For example, during the training phase of a pedestrian recognition model, the ArcFace Loss function, as used in existing technologies, can be employed to calculate the classification loss of the network branch comprised of the first feature processing module 510 and the first classification mapping module 550. In some embodiments of this application, the ArcFace Loss function can be expressed as:

[0091]

[0092] Where N represents the total number of mask images used as training samples, c represents the total number of categories matched by the classification results, and y i f represents the true class value of the i-th training sample. i This represents the predicted classification result for the i-th training sample. The model loss defined in this way allows this network branch to learn features with larger inter-class distances and smaller intra-class distances, thereby improving feature representation capabilities.

[0093] For example, during the training phase of the pedestrian recognition model, the Triplet Loss function in existing technologies can be used to calculate the classification loss of the network branch consisting of the second feature processing module 520, the spatial rotation module 540, and the second classification mapping module 560. In some embodiments of this application, the Triplet Loss function can be expressed as: L1 = max(d(a,p) - d(a,n) + margin, 0); where d(a,p) represents the distance between the mask image a, which serves as the anchor example, and the mask image p, which belongs to the same category as it (i.e., the positive example); d(a,n) represents the distance between the mask image a, which serves as the anchor example, and the mask image n, which belongs to a different category (i.e., the negative example); and margin represents the boundary value. The distance between the mask images can be represented by the similarity distance between the fused features of the mask images.

[0094] Furthermore, by applying and porting the trained pedestrian recognition model, the first classification mapping module 550, the spatial rotation module 540, and the second classification mapping module 560 are removed, and a convolutional layer with a 1*1 convolutional kernel is added. The features output by the first feature processing module 510 and the second feature processing module 520 are fused, and the resulting pedestrian recognition model set is used in pedestrian recognition tasks.

[0095] Step 140: Identify the pedestrian based on the fused features.

[0096] After obtaining the fusion features of the target image, the similarity distance between the obtained fusion features and the template features of each registered person in the pre-created pedestrian database can be calculated to determine the matching degree between each registered person in the pedestrian database and the pedestrian in the target image, thereby determining the pedestrian recognition result. For example, the registered person whose template feature has the smallest similarity distance with the fusion features can be identified as the pedestrian in the target image.

[0097] For a detailed implementation of calculating the similarity distance between the obtained fusion features and the template features of each registered person in the pre-created pedestrian database, please refer to the prior art, and it will not be repeated in the embodiments of this application.

[0098] The pedestrian recognition method disclosed in this application involves acquiring the contour information of pedestrians included in a target image; generating a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image, excluding the human head, is masked; extracting head features from the mask image using a first feature processing module, and extracting human morphological features from the mask image using a second feature processing module; then fusing the features obtained from the head feature extraction and the human morphological feature extraction to obtain fused features; and recognizing the pedestrian based on the fused features, which helps improve the accuracy of pedestrian recognition.

[0099] The pedestrian recognition method disclosed in this application first performs masking processing on the target image to reduce the interference of invalid information introduced by the background and the pedestrian's clothing. Then, it further extracts the pedestrian's head features and human morphological features from the masked image, and then fuses these two features for pedestrian recognition. This helps to improve the accuracy of pedestrian recognition and can effectively avoid the situation where the same pedestrian is identified as different pedestrians due to changing clothing.

[0100] Furthermore, by employing a model structure suitable for extracting facial features to build the first feature processing module for extracting human head features, and by employing a model structure suitable for extracting contour features to build the second feature processing module for extracting human morphological features, each feature processing module can extract its own key features, thereby improving the expressive power of the extracted features for pedestrians.

[0101] On the other hand, the key points of a face have fixed features that do not change significantly over a considerable period of time. By using a model structure suitable for extracting facial features, the first feature processing module extracts features from the mask image, enabling the first feature processing module to fully extract facial features from the human head image for pedestrian recognition, thereby improving the accuracy of pedestrian recognition.

[0102] Example 2

[0103] This application discloses a pedestrian recognition device, such as... Figure 7 As shown, the device includes:

[0104] The pedestrian contour information acquisition unit 710 is used to acquire the contour information of pedestrians included in the target image;

[0105] The mask image generation unit 720 is used to generate a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image other than the human head is masked.

[0106] The feature extraction unit 730 is used to extract head features from the mask image using a first feature processing module, and to extract human body morphology features from the mask image using a second feature processing module. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features.

[0107] The pedestrian identification unit 740 is used to identify the pedestrian based on the fused features.

[0108] In some embodiments of this application, the mask image generation unit 730 is further configured to:

[0109] Based on the contour information, determine the human-shaped image region in the target image where one of the pedestrians is located;

[0110] Determine the human head sub-region within the human-shaped image region;

[0111] A mask image is generated based on the image content of the human head sub-region in the target image and the morphological information of the human figure image region;

[0112] Wherein, the morphological information of the human-shaped image region included in the mask image is consistent with the morphological information of the human-shaped image region in the target image; and, the image pixel values ​​outside the human-shaped image region in the mask image are first color values, the image pixel values ​​within the human-shaped image region but outside the human head sub-region in the mask image are second color values, and the image content of the human head sub-region within the human-shaped image region in the mask image is consistent with the image content of the human head sub-region in the target image.

[0113] In some embodiments of this application, the first feature processing module is used to extract head features from the mask image, and the second feature processing module is used to extract human body morphology features from the mask image. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features, including:

[0114] The first feature processing module in the pre-trained pedestrian recognition model extracts head features from the mask image to obtain the first feature of the target image; and the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the target image; wherein the first feature processing module and the second feature processing module are built based on different neural network structures.

[0115] The first feature and the second feature are fused to obtain a fused feature.

[0116] In some embodiments of this application, such as Figure 8 As shown, the device further includes:

[0117] The pedestrian recognition model training unit 700 is used to train the pedestrian recognition model based on several masked images with category labels; wherein, the masked images include human images, the human head image of the human image is a color image, the image pixel values ​​of the human image in the masked image other than the human head image area are second color values, and the image pixel values ​​of the image area outside the human image in the masked image are first color values.

[0118] In some embodiments of this application, training the pedestrian recognition model based on several masked images with category labels includes:

[0119] For each of the masked images, the following feature extraction and classification operations are performed:

[0120] The first feature processing module in the pedestrian recognition model extracts head features from the mask image to obtain the first feature of the mask image; and the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the mask image.

[0121] The second feature is subjected to a spatial rotation transformation at a random angle to obtain the third feature of the mask image;

[0122] The first feature is classified and mapped to obtain a first classification result prediction value of the mask image, and the third feature is classified and mapped to obtain a second classification result prediction value of the mask image.

[0123] Based on the first classification result prediction value, the second classification result prediction value, and the category label of the corresponding mask image, the classification loss of the pedestrian recognition model is calculated, and the pedestrian recognition model is trained with the goal of optimizing the classification loss.

[0124] In some embodiments of this application, obtaining the contour information of pedestrians included in the target image includes:

[0125] A pre-trained human segmentation model is used to segment the target image into human figures, thereby obtaining the contour information of pedestrians included in the target image; wherein, the human segmentation model is trained using the following method:

[0126] For each sample image, the human figure segmentation model extracts the edge contour features of the sample image through the built-in top-level convolutional layer, and extracts the deep semantic features of the sample image through the built-in bottom-level convolutional layer.

[0127] The edge contour features and the deep semantic features are fused together, and feature mapping is performed based on the fused features to obtain a contour information feature map.

[0128] With the goal of optimizing the contour information feature map, the human figure segmentation model is iteratively trained.

[0129] The pedestrian identification device disclosed in this application is used to implement the pedestrian identification method described in Embodiment 1 of this application. The specific implementation methods of each module of the device will not be repeated here, but can be found in the specific implementation methods of the corresponding steps in the method embodiment.

[0130] The pedestrian recognition device disclosed in this application acquires the contour information of a pedestrian included in a target image; generates a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image, excluding the human head, is masked; a first feature processing module extracts head features from the mask image, and a second feature processing module extracts human morphological features from the mask image; then, the features obtained from the head feature extraction and the human morphological feature extraction are fused to obtain fused features; and the pedestrian is recognized based on the fused features, which helps to improve the accuracy of pedestrian recognition.

[0131] The pedestrian recognition device disclosed in this application first performs masking processing on the target image to reduce interference from invalid information introduced by the background and the pedestrian's clothing. Then, it further extracts the pedestrian's head features and human morphological features from the masked image, and then fuses these two features for pedestrian recognition. This helps to improve the accuracy of pedestrian recognition and can effectively avoid the situation where the same pedestrian is identified as different pedestrians due to changing clothing.

[0132] Furthermore, by employing a model structure suitable for extracting facial features to build the first feature processing module for extracting human head features, and by employing a model structure suitable for extracting contour features to build the second feature processing module for extracting human morphological features, each feature processing module can extract its own key features, thereby improving the expressive power of the extracted features for pedestrians.

[0133] On the other hand, the key points of a face have fixed features that do not change significantly over a considerable period of time. By using a model structure suitable for extracting facial features, the first feature processing module extracts features from the mask image, enabling the first feature processing module to fully extract facial features from the human head image for pedestrian recognition, thereby improving the accuracy of pedestrian recognition.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0135] The present application provides a detailed description of a pedestrian identification method and apparatus. Specific examples have been used to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only for the purpose of helping to understand the method and its core idea. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the idea of ​​the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

[0136] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0138] For example, Figure 9 An electronic device is shown that can implement the methods according to this application. The electronic device can be a PC, mobile terminal, personal digital assistant, tablet computer, etc. The electronic device conventionally includes a processor 910 and a memory 920, and program code 930 stored on the memory 920 and executable on the processor 910. When the processor 910 executes the program code 930, it implements the methods described in the above embodiments. The memory 920 can be a computer program product or a computer-readable medium. The memory 920 can be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 920 has a storage space 9201 for the program code 930 of a computer program for performing any of the method steps described above. For example, the storage space 9201 for the program code 930 can include various computer programs for implementing the various steps in the above methods. The program code 930 is computer-readable code. These computer programs can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The computer program includes computer-readable code that, when executed on an electronic device, causes the electronic device to perform the method according to the above embodiments.

[0139] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the pedestrian recognition method as described in Embodiment 1 of this application.

[0140] Such a computer program product can be a computer-readable storage medium, which can have the same characteristics as... Figure 9The memory 920 in the illustrated electronic device is similarly arranged as storage segments, storage spaces, etc. Program code can be stored, for example, in a compressed form on the computer-readable storage medium. The computer-readable storage medium is typically as shown in the reference... Figure 10 The portable or fixed storage unit is described above. Typically, the storage unit includes computer-readable code 930', which is code read by a processor and, when executed by the processor, implements the various steps of the method described above.

[0141] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0142] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0143] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A pedestrian recognition method, characterized in that, include: Obtain the outline information of pedestrians included in the target image; A mask image is generated based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image other than the human head is masked. The mask image is processed by a first feature processing module to extract head features, and by a second feature processing module to extract human body morphology features. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features. The first feature processing module and the second feature processing module are built based on different neural network structures. The pedestrian is identified based on the fused features; The process involves using a first feature processing module to extract head features from the masked image, and a second feature processing module to extract human body morphology features from the masked image. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features, including: For each of the masked images, the following feature extraction and classification operations are performed: The first feature processing module in the pedestrian recognition model extracts head features from the mask image to obtain the first feature of the mask image; and the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the mask image. The second feature is subjected to a spatial rotation transformation at a random angle to obtain the third feature of the mask image; The first feature is classified and mapped to obtain a first classification result prediction value of the mask image, and the third feature is classified and mapped to obtain a second classification result prediction value of the mask image. Based on the first classification result prediction value, the second classification result prediction value, and the category label of the corresponding mask image, the classification loss of the pedestrian recognition model is calculated, and the pedestrian recognition model is trained with the goal of optimizing the classification loss.

2. The method according to claim 1, characterized in that, The step of generating a mask image based on the contour information and the target image includes: Based on the contour information, determine the human-shaped image region in the target image where one of the pedestrians is located; Determine the human head sub-region within the human-shaped image region; A mask image is generated based on the image content of the human head sub-region in the target image and the morphological information of the human figure image region; Wherein, the morphological information of the human-shaped image region included in the mask image is consistent with the morphological information of the human-shaped image region in the target image; and, the image pixel values ​​outside the human-shaped image region in the mask image are first color values, the image pixel values ​​within the human-shaped image region but outside the human head sub-region in the mask image are second color values, and the image content of the human head sub-region within the human-shaped image region in the mask image is consistent with the image content of the human head sub-region in the target image.

3. The method according to claim 1, characterized in that, The process involves using a first feature processing module to extract head features from the masked image, and a second feature processing module to extract human body morphology features from the masked image. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features, including: The first feature processing module in the pre-trained pedestrian recognition model extracts head features from the masked image to obtain the first feature of the target image. The second feature processing module in the pedestrian recognition model extracts human morphological features from the masked image to obtain the second feature of the target image. The first feature and the second feature are then fused to obtain the fused feature.

4. The method according to claim 3, characterized in that, Before the first feature processing module in the pre-trained pedestrian recognition model extracts head features from the mask image to obtain the first feature of the target image, and the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the target image, the method further includes: The pedestrian recognition model is trained based on several masked images with category labels; wherein the masked images include human figures, the human head image of the human figure is a color image, the image pixel values ​​of the human figure in the masked image other than the human head image area are second color values, and the image pixel values ​​of the image area outside the human figure in the masked image are first color values.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the outline information of pedestrians included in the target image includes: A pre-trained human segmentation model is used to segment the target image into human figures, thereby obtaining the contour information of pedestrians included in the target image; wherein, the human segmentation model is trained using the following method: For each sample image, the human figure segmentation model extracts the edge contour features of the sample image through the built-in top-level convolutional layer, and extracts the deep semantic features of the sample image through the built-in bottom-level convolutional layer. The edge contour features and the deep semantic features are fused together, and feature mapping is performed based on the fused features to obtain a contour information feature map. With the goal of optimizing the contour information feature map, the human figure segmentation model is iteratively trained.

6. A pedestrian recognition device, characterized in that, include: The pedestrian contour information acquisition unit is used to acquire the contour information of pedestrians included in the target image; A mask image generation unit is used to generate a mask image based on the contour information and the target image, wherein the mask image includes the image of the pedestrian in the target image, and the image area of ​​the pedestrian in the mask image other than the human head is masked. The feature extraction unit is used to extract head features from the mask image using a first feature processing module, and to extract human body morphology features from the mask image using a second feature processing module. Then, the features obtained from the head feature extraction and the human body morphology feature extraction are fused to obtain fused features. The first feature processing module and the second feature processing module are built based on different neural network structures. A pedestrian recognition unit is used to identify the pedestrian based on the fused features; The process involves using a first feature processing module to extract head features from the masked image, and a second feature processing module to extract human morphological features from the masked image. Then, the features obtained from the head feature extraction and the human morphological feature extraction are fused to obtain fused features, including: For each of the masked images, the following feature extraction and classification operations are performed: The first feature processing module in the pedestrian recognition model extracts head features from the mask image to obtain the first feature of the mask image; and the second feature processing module in the pedestrian recognition model extracts human body morphology features from the mask image to obtain the second feature of the mask image. The second feature is subjected to a spatial rotation transformation at a random angle to obtain the third feature of the mask image; The first feature is classified and mapped to obtain a first classification result prediction value of the mask image, and the third feature is classified and mapped to obtain a second classification result prediction value of the mask image. Based on the first classification result prediction value, the second classification result prediction value, and the category label of the corresponding mask image, the classification loss of the pedestrian recognition model is calculated, and the pedestrian recognition model is trained with the goal of optimizing the classification loss.

7. The apparatus according to claim 6, characterized in that, The mask image generation unit is further used for: Based on the contour information, determine the human-shaped image region in the target image where one of the pedestrians is located; Determine the human head sub-region within the human-shaped image region; A mask image is generated based on the image content of the human head sub-region in the target image and the morphological information of the human figure image region; Wherein, the morphological information of the human-shaped image region included in the mask image is consistent with the morphological information of the human-shaped image region in the target image; and, the image pixel values ​​outside the human-shaped image region in the mask image are first color values, the image pixel values ​​within the human-shaped image region but outside the human head sub-region in the mask image are second color values, and the image content of the human head sub-region within the human-shaped image region in the mask image is consistent with the image content of the human head sub-region in the target image.

8. An electronic device, comprising a memory, a processor, and program code stored in the memory and executable on the processor, characterized in that, When the processor executes the program code, it implements the pedestrian recognition method according to any one of claims 1 to 5.

9. A computer-readable storage medium having program code stored thereon, characterized in that, When the program code is executed by the processor, it implements the steps of the pedestrian recognition method according to any one of claims 1 to 5.

Images