Pedestrian re-identification method, computer equipment and program product
By extracting and fusing image features in the pedestrian detection block diagram, the problem of low pedestrian recognition accuracy under heterogeneous cameras is solved, and higher autonomous driving safety is achieved.
Patent Information
- Application Number
- CN202411171906.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art has low pedestrian recognition accuracy under vehicle-mounted heterogeneous cameras, which affects the safety of autonomous driving. It is mainly due to the difficulty of feature extraction and matching caused by differences in internal and external parameters, lens distortion and different resolutions of heterogeneous cameras.
By extracting image features from pedestrian detection block diagram, key point extraction and feature fusion are performed, combined with pre-trained models, the full utilization of image features under heterogeneous cameras is achieved, the impact of lens distortion and resolution is reduced, and the accuracy of pedestrian re-identification is improved. sex.
The pedestrian re-identification accuracy under heterogeneous cameras is improved, the safety of autonomous driving is enhanced, and error information interference is reduced and accurate identification of pedestrians is improved.
Smart Images

Figure CN119068415A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to a pedestrian re-identification method, a computer device, and a program product. Background Art
[0002] Person re-identification (Re-ID), also known as pedestrian re-recognition, is a technology that uses computer vision technology to determine whether the same pedestrian exists in images captured by heterogeneous cameras. Due to the differences in parameters between different camera devices in heterogeneous cameras, and the fact that pedestrian appearance is easily affected by clothing, scale, occlusion, posture, and viewing angle, pedestrian re-identification has become a research hotspot in the field of computer vision.
[0003] Pedestrian re-identification based on images taken by vehicle-mounted heterogeneous cameras is an important issue in autonomous driving tasks. The accuracy of recognition will affect the decision-making and safety of autonomous driving. However, some common pedestrian re-identification methods have low re-identification accuracy and poor effect under vehicle-mounted heterogeneous cameras, which affects the safety of autonomous driving. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a pedestrian re-identification method, a computer device, and a program product.
[0005] In a first aspect, an embodiment of the present disclosure provides a pedestrian re-identification method, including:
[0006] Acquire images including pedestrians respectively captured by vehicle-mounted heterogeneous cameras, perform pedestrian detection on the images, and crop the images according to pedestrian detection frames to obtain multiple pedestrian detection frame diagrams;
[0007] Using the pre-trained first model, determining the image features corresponding to each of the pedestrian detection block diagrams, and inputting the image features into the pre-trained second model to extract human key points, and determining the key point features of each human key point area in each of the pedestrian detection block diagrams and the confidence corresponding to the key point features;
[0008] Performing feature fusion on the key point features, the confidence level and the image features to obtain fusion features corresponding to each key point area of the human body in each pedestrian detection frame diagram;
[0009] According to the fusion features of each of the human body key point areas in the pedestrian detection block diagrams respectively corresponding to the heterogeneous cameras, a pedestrian recognition result corresponding to each of the pedestrian detection block diagrams is determined.
[0010] In this implementation, since the internal parameters, focus, and viewing angle of the heterogeneous cameras are different, the images taken by the heterogeneous cameras belong to images of different styles. By extracting image features from the pedestrian detection frame diagram, the feature extraction capability of the first model under different image styles can be fully utilized, and more abundant image details and features can be extracted from the pedestrian detection frame diagram. Then, by extracting key points through image features, key point features with finer granularity can be extracted, and the problems of local features being blocked due to different internal and external parameters of the heterogeneous cameras and texture details of the same pedestrian being changed due to lens distortion of the heterogeneous cameras can be reduced, which have an impact on pedestrian re-identification. Then, by fusing key point features with image features, it is possible to take into account both the global features and local features (i.e., local key point features) of the pedestrian detection frame diagram, avoid interference of erroneous local information and different resolutions of images taken by heterogeneous cameras on pedestrian re-identification, and obtain fused features that can accurately characterize each key point area of the pedestrian. Furthermore, by utilizing the fusion features of key point areas corresponding to different images taken by heterogeneous cameras for pedestrian re-identification, the accuracy of pedestrian re-identification under heterogeneous cameras can be improved, thereby increasing the safety of autonomous driving.
[0011] In a possible implementation manner, the image features include respective regional features of a plurality of regional images corresponding to the pedestrian detection frame diagram;
[0012] The method of using the pre-trained first model to determine the image features corresponding to each of the pedestrian detection block diagrams, and inputting the image features into the pre-trained second model to extract human key points, and determining the key point features of each human key point area in each of the pedestrian detection block diagrams and the confidence corresponding to the key point features, includes:
[0013] Determine, by using the first model, respective regional features of a plurality of regional images corresponding to each of the pedestrian detection frame images;
[0014] Inputting the regional features of the multiple regional images of each of the pedestrian detection frame diagrams into the second model to extract human key points, and determining the key point features of each human key point region in the multiple regional images of each of the pedestrian detection frame diagrams and the confidence corresponding to the key point features;
[0015] The step of fusing the key point features, the confidence level and the image features to obtain fusion features corresponding to each key point region of the human body in each pedestrian detection frame diagram includes:
[0016] The key point features, the confidence levels and the regional features are subjected to feature fusion to obtain fusion features corresponding to the key point regions of the human body in the plurality of regional images of each pedestrian detection frame diagram.
[0017] This implementation method can fully utilize the feature extraction capability of the first model under different image styles by extracting regional features of images of different regions from the pedestrian detection frame diagram, and extract richer image details and features in a small range of image regions. Then, by using the regional features of the regional image to extract key points, it is possible to extract key point features with finer granularity in a small range of images. By fusing the key point features and regional features, it is possible to take into account both the global features and the local features (i.e., local key point features) of the regional image, avoid interference of erroneous local information and different resolutions of images taken by heterogeneous cameras on pedestrian re-identification, and obtain fused features that can accurately characterize each key point region of the pedestrian.
[0018] In a possible implementation, the using the first model to determine the respective regional features of the plurality of regional images corresponding to each of the pedestrian detection frame images includes:
[0019] Using the first model, the pedestrian detection frame diagram is divided into a plurality of region images, and a target sub-region image is intercepted from each of the region images;
[0020] Feature extraction is performed on the multiple target sub-region images to obtain region features corresponding to the multiple region images.
[0021] In this implementation, by dividing the pedestrian detection frame into regions, targeted recognition processing can be achieved for each divided regional image, thereby avoiding the mutual influence between features extracted from different regional images. By further intercepting the regional image, the influence of redundant information in the regional image on pedestrian re-identification can be effectively eliminated, so that the feature extraction process is more focused on the features of the pedestrian itself, thereby improving the accuracy of the extracted regional features.
[0022] In a possible implementation, the step of inputting the regional features of the multiple regional images of each of the pedestrian detection frame images into the second model to extract human key points, and determining the key point features of each human key point region in the multiple regional images of each of the pedestrian detection frame images and the confidence corresponding to the key point features includes:
[0023] The second model of regional features of multiple regional images of each of the pedestrian detection frame diagrams is used to perform at least one convolution process on each of the regional features to obtain convolution features, and the convolution features are used to extract human key points to obtain key point features of each human key point region corresponding to each of the regional images;
[0024] Convolution processing is performed on each of the key point features at least once, and the processed features are normalized to obtain the confidence corresponding to each of the key point features.
[0025] In this implementation, the trained second model has reliable recognition accuracy, so by using the trained second model to extract key points and determine confidence based on regional features, accurate key point features in the regional image and accurate confidence levels corresponding to the key point features can be obtained.
[0026] In a possible implementation, the key point features, the confidence level and the regional features are subjected to feature fusion to obtain fusion features corresponding to the key point regions of the human body in the plurality of regional images of each pedestrian detection frame diagram, including:
[0027] For each of the human key point regions in each of the regional images, weighted processing is performed on the key point features and the confidence of the human key point region, and feature fusion is performed on the weighted result and the key point features to obtain a first feature;
[0028] Performing feature connection processing on the regional feature and the first feature to obtain a second feature;
[0029] The convolutional attention module is used to perform feature fusion on the second feature and the regional feature to obtain the fused feature of the human body key point area.
[0030] In this implementation, by weighting the key point features and confidence corresponding to each key point area, it is possible to penalize the key point features corresponding to the occluded key point area, thereby avoiding the influence of the occlusion position on the recognition accuracy. In addition, by fusing the first feature obtained after weighting processing with the regional feature corresponding to the regional image, and then using the fused feature for pedestrian re-identification, it is possible to simultaneously take into account the global feature similarity and local feature similarity of the regional images corresponding to different cameras, thereby improving the recognition accuracy.
[0031] In a possible implementation, the pedestrian recognition result includes an identification mark of the pedestrian;
[0032] Determining the pedestrian recognition result corresponding to each pedestrian detection frame diagram according to the fusion features of each human key point area in the pedestrian detection frame diagram corresponding to the heterogeneous cameras respectively includes:
[0033] Determining a reference camera and a matching camera among the heterogeneous cameras;
[0034] For any first pedestrian detection frame image corresponding to the reference camera, determine the similarity between the first pedestrian detection frame image and the second pedestrian in each of the second pedestrian detection frame images according to the fusion features corresponding to the human body key point regions in the first pedestrian detection frame image and the fusion features of the human body key point regions in each second pedestrian detection frame image corresponding to the matching camera;
[0035] Determining a matching result between the first pedestrian and each of the second pedestrians according to the similarity; the matching result is used to indicate whether the second pedestrian is the same as the first pedestrian;
[0036] According to the matching result of the first pedestrian in each first pedestrian detection frame corresponding to the reference camera, the identification mark of each first pedestrian and the identification mark of each second pedestrian are determined.
[0037] In this implementation, between different images taken by heterogeneous cameras, matching of key points at corresponding positions is performed by using fusion features obtained by fusing regional features and key point features corresponding to each key point area, thereby obtaining matching results between multiple key point areas in different images, and then matching pedestrians using the matching results of multiple key point areas, thereby improving the accuracy of the matching results and obtaining an accurate identification mark corresponding to each pedestrian.
[0038] In a possible implementation, the heterogeneous camera includes a wide-angle camera and a fisheye camera;
[0039] Before acquiring the images including pedestrians respectively collected by the vehicle-mounted heterogeneous cameras, the method further includes:
[0040] Acquire a first image including pedestrians taken by the wide-angle camera and a second image including pedestrians taken by the fisheye camera;
[0041] Performing a dedistortion process on the second image to obtain a third image corresponding to the fisheye camera;
[0042] The performing pedestrian detection on the image and cropping the image according to the pedestrian detection frame to obtain a plurality of pedestrian detection frame diagrams includes:
[0043] Pedestrian detection is performed on the first image and the third image respectively, and the first image and the third image are cropped respectively according to the pedestrian detection frame to obtain multiple pedestrian detection frame diagrams.
[0044] In this implementation, since the image taken by the fisheye camera has severe distortion, the influence of the distortion can be eliminated to a certain extent by dedistorting the second image taken by the fisheye camera. The dedistorted image is then used for pedestrian detection, which can improve the accuracy of the determined pedestrian detection block diagram.
[0045] In a possible implementation manner, the first model is trained according to the following steps:
[0046] Acquire a sample image data set; the sample image data set includes a plurality of sample image pairs, and the sample image pairs include sample images respectively taken by the heterogeneous cameras;
[0047] For any of the sample image pairs, a sample pedestrian detection block diagram in each sample image included in the sample image pair is input into a first model to be trained to obtain prediction region features corresponding to a plurality of sample region images in the sample pedestrian detection block diagram;
[0048] Merging the predicted region features and the manually constructed shared features corresponding to the sample region image, and inputting the merged features into a decoding module in the first model to be trained to obtain a restored region image corresponding to the sample region image;
[0049] Determine a first loss of a first model to be trained according to each of the sample region images corresponding to each of the sample image pairs and the restored region image corresponding to each of the sample region images;
[0050] The first model to be trained is iteratively trained using the first loss until a training cutoff condition is met, thereby obtaining a trained first model.
[0051] In this implementation, the predicted region features extracted by the coding model are decoded by the decoding module, and the restored region image corresponding to the predicted region features extracted by the coding model can be restored. Then, the first loss is constructed by the restored region image and the original sample region image, so that the first loss can be used to accurately characterize the extraction error of the coding model when extracting regional features. Finally, the first model to be trained is iteratively trained using the first loss, which can effectively improve the regional feature extraction capability of the first model. In addition, by training with sample images collected by heterogeneous cameras, the first model can be focused on pedestrian recognition under images of different styles, thereby improving the feature extraction capability of the first model under heterogeneous cameras.
[0052] In a possible implementation, the method further includes the step of training the second model:
[0053] Obtaining sample area features of each sample area image in a plurality of sample pedestrian detection frame diagrams; the sample area features are obtained by extracting features of the sample pedestrian detection frame diagram in the sample image using a trained first model; the sample image is an image in a sample image pair; the sample image pair includes sample images respectively taken by the heterogeneous cameras;
[0054] Input the sample area features into the second model to be trained to extract human key points, determine the predicted key point features of each human key point area corresponding to each sample area image in each sample pedestrian detection frame diagram and the prediction confidence corresponding to the predicted key point features;
[0055] Determining the prediction category of each pixel point in each of the sample area images by using the prediction key point features and the prediction confidence;
[0056] Performing feature fusion on the predicted key point features, the predicted confidence and the sample area features to obtain predicted fusion features corresponding to each human key point area in each sample area image in each sample pedestrian detection frame diagram, and determining the predicted identification of the pedestrian in each sample pedestrian detection frame diagram according to the predicted fusion features;
[0057] According to the predicted category and label category of each pixel point in the sample pedestrian detection frame diagram, the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram, and the predicted fusion feature, a second loss of the second model to be trained is determined, and the second model to be trained is iteratively trained using the second loss until a training cutoff condition is met to obtain a trained second model.
[0058] In this implementation, the second loss determined by using the predicted category and label category of the pixel point, and the predicted identification and label identification of the pedestrian can accurately characterize the key point feature extraction loss and identification prediction loss of the second model. Iterative training of the second model using the second loss can effectively improve the key point feature extraction capability and identification prediction capability of the second model.
[0059] In a possible implementation, determining the second loss of the second model to be trained according to the predicted category and label category of each pixel in the sample pedestrian detection frame map, the predicted identifier and label identifier of the pedestrian in the sample pedestrian detection frame map, and the predicted fusion feature includes:
[0060] Determining a key point prediction loss according to a prediction category and a label category of each pixel point in the sample pedestrian detection frame map;
[0061] Determining recognition loss according to the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram and the predicted fusion feature;
[0062] The second loss is determined according to the key point prediction loss and the recognition loss.
[0063] This implementation method uses the prediction category and label category to accurately determine the loss of the second model when performing key point extraction (i.e., key point prediction loss). Use the prediction identifier and label identifier, as well as the prediction fusion feature, to accurately determine the recognition loss of the second model when performing pedestrian matching and recognition.
[0064] In a possible implementation, determining the recognition loss according to the predicted identification and label identification of the pedestrian in the sample pedestrian detection block diagram and the predicted fusion feature includes:
[0065] Determine a marker prediction loss according to the predicted markers of the pedestrians in each of the sample pedestrian detection frame diagrams, the label marker of each of the pedestrians, and the first number of the pedestrians;
[0066] For any first sample pedestrian detection frame diagram in each of the sample pedestrian detection frame diagrams, according to a first label identifier of a pedestrian in the first sample pedestrian detection frame diagram and a second label identifier of a pedestrian in each second sample pedestrian detection frame diagram except the first sample pedestrian detection frame diagram, determining a positive sample set and a negative sample set corresponding to the first sample pedestrian detection frame diagram from the second sample pedestrian detection frame diagram;
[0067] Determine the triplet loss by using the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the positive sample set, and the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the negative sample set;
[0068] The recognition loss is determined according to the identification prediction loss and the triplet loss.
[0069] In this implementation, the identification prediction loss can accurately characterize the difference of the second model when performing identification prediction, and the triplet loss can characterize the difference between the predicted fusion features output by the second model and the predicted fusion features corresponding to the positive sample and the negative sample, respectively. Therefore, by using the identification prediction loss and the triplet loss together as the recognition loss to train the second model, the processing capability of the second model when performing feature fusion and identification output can be improved.
[0070] In a possible implementation, determining a positive sample set and a negative sample set corresponding to the first sample pedestrian detection block diagram from the second sample pedestrian detection block diagram according to the first label identification of the pedestrian in the first sample pedestrian detection block diagram and the second label identification of the pedestrian in each second sample pedestrian detection block diagram except the first sample pedestrian detection block diagram, includes:
[0071] Selecting a second number of target pedestrians from the first number of pedestrians;
[0072] For each of the target pedestrians, according to the third label identifier of the target pedestrian and each of the second label identifiers, a third number of third sample pedestrian detection frame diagrams corresponding to each of the target pedestrians are screened out from each of the second sample pedestrian detection frame diagrams;
[0073] According to the first label identification and the third label identification of each target pedestrian, a set of positive samples with the same identification as that corresponding to the first sample pedestrian detection frame diagram and a set of negative samples with different identifications corresponding to the first sample pedestrian detection frame diagram are determined from the third sample pedestrian detection frame diagram corresponding to each target pedestrian.
[0074] This implementation method, by limiting the number of target pedestrians and the number of sample pedestrian detection frame diagrams for each target pedestrian, can limit the number of training samples in a batch, thereby avoiding the problem of high resource requirements and reduced model generalization ability caused by a batch size that is too large, and avoiding the problem of slow training speed and slow convergence caused by a batch size that is too small.
[0075] In a possible implementation, the use of the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the positive sample set, and the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the negative sample set to determine the triplet loss includes:
[0076] For any second sample pedestrian detection frame diagram in the positive sample set, selecting a plurality of first target features from the predicted fusion features of each human key point area corresponding to the second sample pedestrian detection frame diagram;
[0077] For any second sample pedestrian detection frame diagram in the negative sample set, selecting a plurality of second target features from the predicted fusion features of each human key point area corresponding to the second sample pedestrian detection frame diagram;
[0078] Selecting, from the predicted fusion features of each human key point region corresponding to the first sample pedestrian detection frame diagram, a third target feature matching each of the first target features and a fourth target feature matching each of the second target features;
[0079] The triplet loss is determined according to the Euclidean distance between each of the first target features and the matched third target feature, and the Euclidean distance between each of the second target features and the matched fourth target feature.
[0080] This implementation method uses the predicted key point features (i.e., the first target feature, the second target feature, and the third target feature) corresponding to some key point regions to determine the loss, which can avoid the problem of high computing resource requirements and slow processing speed caused by using too many key points. In addition, by determining the triplet loss through the features at the body part level (i.e., the key point region level), the second model can pay more attention to the invariant identification features between images of different styles, and combined with the identification prediction loss, the second model can identify the identity of pedestrians from a global perspective.
[0081] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.
[0082] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, which, when executed, executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.
[0083] For a description of the effects of the above-mentioned computer device and computer program product, please refer to the description of the above-mentioned pedestrian re-identification method, which will not be repeated here.
[0084] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0086] Figure 1 A flowchart of a pedestrian re-identification method provided by an embodiment of the present disclosure is shown;
[0087] Figure 2 A schematic diagram showing the structure of a key point feature extraction module of a second model provided by an embodiment of the present disclosure is shown;
[0088] Figure 3 A schematic diagram of the structure of a feature fusion module of a second model provided by an embodiment of the present disclosure is shown;
[0089] Figure 4 A schematic diagram of the structure of a first model provided by an embodiment of the present disclosure is shown;
[0090] Figure 5 A schematic diagram of a second loss determination process provided by an embodiment of the present disclosure is shown;
[0091] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure more clear, the technical solution in the embodiments of the present disclosure will be described below in conjunction with the accompanying drawings in the embodiments of the present disclosure.
[0093] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.
[0094] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0095] According to research, pedestrian re-identification can be divided into two processes: one is to identify pedestrians in different images (i.e. pedestrian detection), and the other is to match pedestrians in different images (i.e. pedestrian matching). Existing pedestrian re-identification methods usually use existing high-precision pedestrian detection models to identify pedestrians from images, and then use deep learning methods (such as learning methods based on global features or learning methods based on local features) to perform pedestrian matching to obtain pedestrian re-identification results. However, for heterogeneous cameras, they often have the following problems:
[0096] 1. Different camera intrinsic parameters and distortions, specifically manifested in different focal lengths and viewing angles of different cameras in heterogeneous cameras, and different lens distortions. For example, for common wide-angle cameras and fisheye cameras, wide-angle cameras have short focal lengths and larger viewing angles, which can obtain wider viewing angles. Fisheye cameras have shorter focal lengths and larger viewing angles than wide-angle cameras, and can provide wider shooting angles. In addition, the lens distortion coefficients of the two cameras are different, resulting in large differences in the content of the pictures taken by the two cameras. Therefore, the pictures collected by the two cameras belong to different styles, which greatly hinders the extraction of features for the Re-ID task.
[0097] 2. The different camera extrinsics and installation positions result in large differences in shooting angles between different cameras. For example, in general, wide-angle cameras have lower shooting angles, while fisheye cameras have higher shooting angles, causing two pedestrians that do not overlap under the fisheye lens to appear occluded in the wide-angle lens. When multiple pedestrian features appear in the identified pedestrian detection frame due to occlusion, the features extracted in the pedestrian detection frame are no longer just the unique features of the pedestrian, but are added with interference features of other pedestrians. For example, in a pedestrian detection frame, the full body features of target A and the half-body features of target B appear. After the model extracts the features, it is unable to distinguish between the features that truly represent target A and the features that truly represent target B, which causes great interference to cross-lens target recognition.
[0098] 3. Different resolutions. Specifically, different cameras have different resolutions. Affected by the resolution, the quality of images collected by different cameras is different. When encountering situations where the similarity is high and they are not the same pedestrians, or when encountering situations where some features of two pedestrians are similar, it is difficult to distinguish between different pedestrians.
[0099] Due to the above differences and problems brought by heterogeneous cameras, both the global feature learning method and the local feature learning method have poor pedestrian matching effects under heterogeneous cameras, which in turn affects the accuracy of pedestrian re-identification.
[0100] Based on the above research, the present disclosure provides a pedestrian re-identification method, a computer device and a program product. By extracting image features from a pedestrian detection block diagram, the feature extraction capability of the first model under different image styles can be fully utilized, and more abundant image details and features can be extracted from the pedestrian detection block diagram. Then, by extracting key points through image features, key point features with finer granularity can be extracted, and the local features caused by different internal and external parameters of heterogeneous cameras and the lens distortion of heterogeneous cameras causing the texture details of the same pedestrian to change, etc., can be reduced. The impact on pedestrian re-identification. Then, by fusing key point features and image features, it is possible to take into account the global features and local features (i.e., local key point features) of the pedestrian detection block diagram, avoid the interference of erroneous local information and the different resolutions of images taken by heterogeneous cameras on pedestrian re-identification, and thus obtain fused features that can accurately characterize each key point area of the pedestrian. Furthermore, by using the fused features of the key point areas corresponding to different images taken by heterogeneous cameras for pedestrian re-identification, the accuracy of pedestrian re-identification under heterogeneous cameras can be improved, thereby increasing the safety of autonomous driving.
[0101] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0102] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0103] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0104] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:
[0105] Backbone (VIT) refers to a model that uses Vision Transformer (ViT) as the backbone network. ViT is a model architecture for visual tasks. Its overall idea is to perform tasks such as image classification based on a pure Transformer structure.
[0106] To facilitate understanding of the present embodiment, a pedestrian re-identification method disclosed in the embodiment of the present disclosure is first introduced in detail. The executor of the pedestrian re-identification method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities or various cloud servers deployed in the cloud, wherein the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a vehicle-mounted terminal, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computer device, etc.; in some possible implementation methods, the pedestrian re-identification method may be implemented by a processor calling computer-readable instructions stored in a memory.
[0107] The pedestrian re-identification method provided by the embodiment of the present disclosure is described below by taking a computer device as an example of an execution subject.
[0108] like Figure 1 As shown, it is a flowchart of a pedestrian re-identification method provided by an embodiment of the present disclosure, which may include the following steps:
[0109] S101: Obtain images including pedestrians collected by vehicle-mounted heterogeneous cameras, perform pedestrian detection on the images, and crop the images according to pedestrian detection frames to obtain multiple pedestrian detection frame diagrams.
[0110] Here, heterogeneous cameras are cameras with different camera construction principles, such as fisheye cameras and wide-angle cameras, visible light cameras and infrared cameras, different types of cameras, etc. A vehicle-mounted heterogeneous camera is a heterogeneous camera installed on a vehicle. The vehicle-mounted heterogeneous camera can be used to perform visual perception of the vehicle, thereby assisting tasks such as autonomous driving. In the following text, the pedestrian re-identification method provided by the embodiment of the present disclosure is described by taking the vehicle-mounted heterogeneous cameras as fisheye cameras and wide-angle cameras as examples.
[0111] The above image is an image including pedestrians collected by the heterogeneous camera during the driving process of the vehicle, and the image may include at least one pedestrian. The images taken by different cameras in the heterogeneous camera may include the same pedestrian and / or different pedestrians.
[0112] For example, an image may include images of pedestrians taken by a fisheye camera and a wide-angle camera on a vehicle at the same time. Since the shooting range of a fisheye camera is larger than that of a wide-angle camera, the content of the image taken by the fisheye camera is richer than that of the image taken by the wide-angle camera, and there may be some overlap between the images taken by the fisheye camera and the images taken by the wide-angle camera. For example, the image taken by the fisheye camera includes pedestrians 1 to 3, and the image taken by the wide-angle camera includes pedestrians 1 and 2.
[0113] The pedestrian detection frame is the regional image corresponding to the pedestrian detection frame where the pedestrian in the image is located. At least one pedestrian detection frame can be correspondingly intercepted from an image, and the number of pedestrian detection frames in the images collected by the vehicle-mounted heterogeneous cameras can be the same or different.
[0114] Taking the case where the vehicle-mounted heterogeneous cameras include a fisheye camera and a wide-angle camera, in a specific implementation, the images taken by the fisheye camera and the wide-angle camera can be obtained, and then pedestrian detection can be performed on the images taken by the two cameras respectively to obtain a pedestrian detection frame corresponding to the pedestrian in each image, and then the pedestrian detection frame is used to intercept the image area to obtain at least one pedestrian detection frame diagram corresponding to each image. Among them, the operation of performing pedestrian detection on the image can be performed using any trained pedestrian detection model in the prior art, for example, the YoloV6 model.
[0115] S102: Using the pre-trained first model, determine the image features corresponding to each pedestrian detection frame diagram, and input the image features into the pre-trained second model to extract human key points, determine the key point features of each human key point area in each pedestrian detection frame diagram and the confidence corresponding to the key point features.
[0116] Here, the first model is a model for extracting features from an image, and the first model can be a neural network model constructed using backbone (VIT) as a framework. The second model can be a pedestrian matching model, which can specifically include three parts, one of which is a key point feature extraction module, one of which is a feature fusion module, and another of which is a feature matching module. Among them, the key point feature extraction module is used to extract the features of each human key point area in the image, and each human key point area can be an area belonging to different body parts, and each pixel point in the key point area is recognized as a human key point by the second model. Among them, the human key point areas corresponding to different body parts can include, for example, the head area, the left arm area, the right arm area, the trunk area, the left leg area, the right leg area, the left foot area, the right foot area, etc. The second model can be constructed based on the heatmap key point extraction model. The feature fusion module is used to fuse the features of the human key point area with the image features of the image to which the human key point area belongs; the feature matching module is used to match pedestrians in different images using the features fused by the feature fusion module, and output the pedestrian recognition results corresponding to the pedestrians.
[0117] The key point features of the human key point region can be determined based on the point features of each human key point in the key point region. The image features are used to characterize the overall features of the pedestrian detection frame.
[0118] In specific implementation, for an image captured by any camera in the vehicle-mounted heterogeneous camera, each pedestrian detection block diagram corresponding to the image can be input into a pre-trained first model for feature extraction to obtain image features corresponding to each pedestrian detection block diagram. Then, for any pedestrian detection block diagram, the image features of the pedestrian detection block diagram can be input into a pre-trained second model, and the key point feature extraction module in the second model is used to perform human key point area detection and key point feature extraction to obtain key point features of each human key point area in the pedestrian detection block diagram and the confidence corresponding to the key point features.
[0119] S103: performing feature fusion on key point features, confidence levels and image features to obtain fusion features corresponding to each human key point region in each pedestrian detection frame diagram.
[0120] Here, since the non-human key point region may also contain important information for identifying pedestrians, in order to avoid ignoring the important information in the non-human key point region, the disclosed embodiment can also fuse the key point features with the image features of the pedestrian detection frame map to which the key point region belongs after extracting the key point features, so that the fused features contain both local key point features and global image features. One human key point region corresponds to one fused feature.
[0121] Exemplarily, for the key point features of any key point area of the human body in any pedestrian detection frame diagram, the key point features, confidence, and image features of the pedestrian detection frame diagram can be input into the feature fusion module in the second model. Then, the feature fusion module is used to determine whether the confidence of the key point feature is greater than the preset confidence. If so, the key point features and the image features of the pedestrian detection frame diagram can be fused using spatial attention and channel attention to obtain the fused features corresponding to the key point area of the human body. If not, the key point features can be weighted using the confidence of the key point features, and then the weighted results and the image features of the pedestrian detection frame diagram can be fused using spatial attention and channel attention to obtain the fused features corresponding to the key point area of the human body.
[0122] S104: Determine a pedestrian recognition result corresponding to each pedestrian detection frame diagram according to fusion features of each human body key point area in the pedestrian detection frame diagrams corresponding to the heterogeneous cameras.
[0123] Here, the pedestrian recognition result may be an Identity Document (ID) for each pedestrian.
[0124] In specific implementation, based on the above S101 to S103, the fusion features of each human key point area in each pedestrian detection frame included in the image taken by each camera in the heterogeneous camera (i.e., the vehicle-mounted heterogeneous camera) can be obtained. Then, for any camera in the heterogeneous camera, the feature matching module in the second model can be used to perform feature matching on the fusion features of each human key point area in each pedestrian detection frame in the image taken by the camera and the fusion features of each human key point area in each pedestrian detection frame in the image taken by another camera in the heterogeneous camera, and based on the feature matching results, determine the pedestrian recognition results of the pedestrians in each pedestrian detection frame in the two images. Among them, when performing feature matching on the fusion features of the human key point areas in the two pedestrian detection frame diagrams, the fusion features of the human key point areas corresponding to the same body part can be matched. For example, the pedestrian detection frame Figure 1 The fusion features of the head key point area in the image are combined with the pedestrian detection frame Figure 2 The fusion features of the head key point area in the pedestrian detection frame are matched Figure 1 The fusion features of the arm key point area in the image are combined with the pedestrian detection frame Figure 2 The fusion features of the arm key point area are matched.
[0125] For example, the heterogeneous cameras include camera 1 and camera 2. The image captured by camera 1 includes a pedestrian detection frame. Figures 1 to 3 , pedestrian detection box Figures 1 to 3 Corresponding to pedestrians 1 to 3, the image captured by camera 2 includes pedestrian detection frames Figure 4 ~7, pedestrian detection frame Figure 4 ~7 correspond to pedestrians 4 to 7. In implementation, the feature matching module in the second model can be used to transform the pedestrian detection frame Figure 1 The fusion features of each human key point area in the pedestrian detection frame are Figure 4 The fusion features of each key point area of the human body in ~7 are matched, and based on the matching results, it is determined whether there is a pedestrian detection frame in pedestrians 4 to 7. Figure 1 In this way, the pedestrian detection frame can be used to Figures 1 to 3 Pedestrian detection box Figure 4 The matching results of ~7 are used to determine the pedestrian recognition results corresponding to pedestrians 1 to 7 respectively.
[0126] In this way, by extracting image features from the pedestrian detection block diagram, the feature extraction capability of the first model under different image styles can be fully utilized, and more abundant image details and features can be extracted from the pedestrian detection block diagram. Then, by extracting key points through image features, key point features with finer granularity can be extracted, and the problems of local feature occlusion caused by different internal and external parameters of heterogeneous cameras and texture detail changes of the same pedestrian caused by lens distortion of heterogeneous cameras can be reduced, which will affect pedestrian re-identification. Then, by fusing key point features with image features, the global features and local features (i.e., local key point features) of the pedestrian detection block diagram can be taken into account, and the interference of erroneous local information and different resolutions of images taken by heterogeneous cameras on pedestrian re-identification can be avoided, so as to obtain fused features that can accurately characterize each key point area of the pedestrian. Furthermore, by using the fused features of key point areas corresponding to different images taken by heterogeneous cameras for pedestrian re-identification, the accuracy of pedestrian re-identification under heterogeneous cameras can be improved, thereby increasing the safety of autonomous driving.
[0127] In one embodiment, the image features may include the respective regional features of the multiple regional images corresponding to the pedestrian detection frame diagram. Here, the multiple regional images may be cropped from the pedestrian detection frame diagram, and the regional features are used to characterize the overall features of the regional images. For the above S102, the following steps may be followed:
[0128] S102-1: Using the first model, determine the respective regional features of the multiple regional images corresponding to each pedestrian detection frame.
[0129] Here, the regional image is an image corresponding to a part of the region in the pedestrian detection frame diagram, and a pedestrian detection frame diagram may include multiple regional images. The image areas corresponding to the multiple regional images are greater than or equal to the area of the pedestrian detection frame diagram. Exemplarily, the multiple regional images corresponding to the pedestrian detection frame diagram may include an upper left regional image, an upper right regional image, a lower left regional image, and a lower right regional image of the pedestrian detection frame diagram. The regional feature is a feature corresponding to the regional image, which may specifically be a feature vector extracted by the first model.
[0130] The first model can be specifically used to divide the pedestrian detection frame diagram into regions and extract the region features corresponding to each divided region. The first model can be specifically a neural network model constructed using backbone (VIT) as a framework.
[0131] In specific implementation, for an image taken by any of the vehicle-mounted heterogeneous cameras, each pedestrian detection frame diagram corresponding to the image can be input into a pre-trained first model, and the pedestrian detection frame diagram is first divided into regions using the first model. Then, feature extraction is performed on the regional images corresponding to each divided region, thereby obtaining the regional features of each of the multiple regional images corresponding to the pedestrian detection frame diagram.
[0132] S102-2: Input the regional features of multiple regional images of each pedestrian detection frame into the second model to extract human key points, determine the key point features of each human key point area in the multiple regional images of each pedestrian detection frame and the confidence corresponding to the key point features.
[0133] Here, the key point feature extraction module in the second model can be specifically used to extract the features of each human key point area in the regional image; the feature fusion module can be specifically used to fuse the features of the human key point area with the regional features of the regional image to which the key point area belongs; the feature matching module can be specifically used to use the features fused by the feature fusion module to match pedestrians in different images, and output the pedestrian recognition results corresponding to the pedestrians.
[0134] The key point feature is the regional feature corresponding to the identified key point region of the human body. For example, the key point feature may be an attention map corresponding to the key point. The confidence of the key point feature may specifically be a confidence score of the key point feature.
[0135] The human body key point regions included in different region images may be completely different or may partially overlap.
[0136] In specific implementation, for any area image in any pedestrian detection frame diagram, the area features of the area image can be input into the key point feature extraction module in the pre-trained second model for key point detection, and the human key point areas corresponding to each body part existing in the area image are detected. For each human key point area detected, the key point features of the area are extracted, and the confidence level corresponding to each key point feature is determined.
[0137] In this way, for any area image in any pedestrian detection frame diagram included in an image taken by any camera, the second model and the area features corresponding to the area image can be used to obtain the key point features and confidence levels of each human key point area included in the area image.
[0138] Furthermore, the above S103 may be implemented according to the following steps:
[0139] The key point features, confidence levels and regional features are fused to obtain the fused features corresponding to each key point region of the human body in multiple regional images of each pedestrian detection frame diagram.
[0140] Here, after extracting the key point features of the key point region of the human body, the key point features and the regional features of the regional image to which the key point region of the human body belongs can be fused, so that the fused features include both local key point features and global regional features. That is, the fused features can characterize the key point features of a certain key point region of the human body and the regional features of the regional image to which the key point region of the human body belongs.
[0141] Exemplarily, for the key point features of any human key point region in any regional image, the key point features, confidence, and regional features of the regional image can be input into the feature fusion module in the second model. Then, the feature fusion module is used to determine whether the confidence of the key point feature is greater than the preset confidence. If so, the key point features and the regional features of the regional image can be fused using spatial attention and channel attention to obtain the fused features corresponding to the human key point region. If not, the key point features can be weighted using the confidence of the key point features, and then the weighted results and the regional features of the regional image can be fused using spatial attention and channel attention to obtain the fused features corresponding to the human key point region.
[0142] Then, for any camera in the heterogeneous cameras, the feature matching module in the second model can be used to perform feature matching on the fused features of each human key point area of each pedestrian detection frame diagram in the image taken by the camera with the fused features of each human key point area of each pedestrian detection frame diagram in the image taken by another camera in the heterogeneous cameras. Based on the feature matching results, the pedestrian recognition results of the pedestrians in each pedestrian detection frame diagram in different images are determined. When performing feature matching on the fused features of the human key point areas in two pedestrian detection frame diagrams, the fused features of the human key point areas of the same body part can be matched. For example, matching the fused features of the human key point areas in the pedestrian detection frame diagrams Figure 1 The fusion features of the head key point area in the upper left area of the image and the pedestrian detection box Figure 2 The fusion features of the head key point area of the upper left area image are matched to form a pedestrian detection frame Figure 1 The fusion features of the arm key point area in the lower right image and the pedestrian detection box Figure 2 The fused features of the arm key point area in the lower right area of the image are matched.
[0143] In this way, by extracting regional features of different regional images from the pedestrian detection frame diagram, the feature extraction capability of the first model under different image styles can be fully utilized, and richer image details and features can be extracted in a small range of image regions. Then, by using the regional features of the regional image for key point extraction, it is possible to extract key point features with finer granularity in a small range of images, and reduce the impact of local features being blocked due to different internal and external parameters of heterogeneous cameras, and the lens distortion of heterogeneous cameras causing changes in texture details of the same pedestrian, which have on pedestrian re-identification. Then, by fusing key point features and regional features, it is possible to take into account both the global features and local features (i.e., local key point features) of the regional image, avoid the interference of erroneous local information and the different resolutions of images taken by heterogeneous cameras on pedestrian re-identification, and obtain fused features that can accurately characterize each key point region of the pedestrian.
[0144] In one embodiment, the step of determining the regional features in S102-1 above may be implemented as follows:
[0145] S102-1-1: Using the first model, divide the pedestrian detection frame into multiple area images, and capture a target sub-area image from each area image.
[0146] For example, the first model can be used to dynamically and real-time divide the pedestrian detection frame into multiple regional images, or the pedestrian detection frame can be divided into multiple regional images using a manually set division method. The dynamic real-time division can, for example, determine the number of divisions based on the size of the pedestrian detection frame, and divide the pedestrian detection frame into multiple regional images according to the number.
[0147] The artificially set division method can be to divide the pedestrian detection frame into four fixed overlapping area images, wherein the four fixed overlapping area images can include the upper left area image of the pedestrian detection frame (denoted as L ul ), the upper right area image (denoted as L ur ), the lower left area image (denoted as L dl ) and the lower right area image (denoted as L dr ). There may be partial overlap between adjacent regional images. For example, for any pedestrian detection frame map, after the pedestrian detection frame map is input into the first model, the first model may divide the pedestrian detection frame map into L ul , L ur , L dl , and L dr .
[0148] After obtaining each regional image, the target sub-region image can be intercepted from each regional image according to the manually set interception ratio. The interception ratio can be set based on experience and is not specifically limited in the embodiment of the present disclosure. For example, the interception ratio can be 25%. The interception method can include random interception or interception from the central area.
[0149] The area of the target sub-region image, and the interception ratio of the area of the regional image. The target sub-region image is the image corresponding to the partial region intercepted from the regional image. The target sub-region image can be the image corresponding to a continuous region, or it can be composed of images corresponding to multiple different regions in the regional image.
[0150] In a specific implementation, in order to prevent the mutual influence between the features extracted from different regional images, after the four regional images are divided, each regional image can be further cropped. Specifically, for any regional image, the first model can select a continuous region of the size of the cropping ratio from the regional image according to the cropping ratio, and crop the region from the regional image to obtain a target sub-region image corresponding to the regional image.
[0151] Alternatively, for any regional image, the first model can first randomly crop local views of different sizes from the regional image to obtain a set of local views corresponding to the regional image. Among them, a localviews can be understood as a sub-regional image in the regional image. Then, the local views of the local views set corresponding to the regional image can be randomly masked according to the cropping ratio, that is, the local views of (1-cropping ratio) are discarded, so as to retain the local views of the cropping ratio, and the retained local views are used as the target sub-regional image corresponding to the regional image. Taking the cropping ratio of 25% as an example, 75% of the local views in the localviews set corresponding to the regional image can be randomly masked, and 25% of the local views are retained as the target sub-regional image.
[0152] In this way, after further cropping the regional image according to the interception ratio, the image used for regional feature extraction can be only 25% of the regional image, thereby largely eliminating the influence of redundant information in the regional image on feature extraction, so that the first model can be more focused on the extraction of pedestrian information. Here, the reason why the first model can be guaranteed to be focused on the extraction of pedestrian information is that in the process of training the first model, as the model continues to learn, the model's ability to extract pedestrian information will be gradually improved, thereby ensuring that the finally trained first model can be more focused on the extraction of pedestrian information.
[0153] S102-1-2: Extract features from multiple target sub-region images to obtain region features corresponding to the multiple region images.
[0154] In specific implementation, after the first model crops out the target sub-region image corresponding to each region image, the target sub-region image corresponding to each region image can be respectively input into the encoder encoder in the first model, and the encoder is used to perform feature extraction and encoding processing on the target sub-region image, so as to obtain the region features corresponding to the region image to which the target sub-region image belongs.
[0155] In one embodiment, the step of extracting key point features in S102 may be implemented as follows A1 and A2:
[0156] A1: Input the regional features of multiple regional images of each pedestrian detection frame into the second model, perform at least one convolution process on each regional feature to obtain a convolution feature, and use the convolution feature to extract human key points to obtain the key point features of each human key point area corresponding to each regional image.
[0157] Here, the human body key point area may include the head key point area, the left arm key point area, the right arm key point area, the torso key point area, the left leg key point area, the right leg key point area, the left foot key point area, the right foot key point area, etc.
[0158] During specific implementation, for any area image in any pedestrian detection block diagram, the area features of the area image can be input into the trained second model, and the convolution layer (conv) of the key point feature extraction module in the second model can perform at least one convolution process on the area features to obtain convolution features. Then, the key point area feature layers corresponding to different human key point areas can be used to perform feature recognition and key point extraction on the convolution features to obtain the key point features corresponding to each human key point area in the area image. For example, the second model may include a head key point area feature layer (head), a left arm / right arm key point area feature layer (left / right arm), a torso key point area feature layer (torso), a left leg / right leg key point area feature layer (left / right leg), and a left foot / right foot key point area feature layer (left / right foot).
[0159] After obtaining the convolutional features corresponding to any regional image, the convolutional features can be input into the corresponding key point regional feature layer for key point feature extraction according to the position of the regional image in the pedestrian detection frame diagram, and the key point features corresponding to the human key point region can be obtained. For example, for the upper left regional image, after obtaining the convolutional features corresponding to the upper left regional image, the convolutional features can be respectively input into the head feature layer, the left arm feature layer and the torso feature layer, and the key point features corresponding to the head key point region output by the head feature layer (for example, head feature map), the key point features corresponding to the left arm key point region (for example, left arm feature map), and the key point features corresponding to the torso key point region (for example, torso feature map) are obtained. It can be understood that if a certain human key point does not exist in the upper left regional image, the key point features output by the corresponding key point regional feature layer can be preset information, such as empty, all 1 or 0 feature maps, etc. For example, in the case where there is no left arm in the upper left regional image, the left arm feature layer can output a left arm feature map that is all 0 or 1.
[0160] Alternatively, after obtaining the convolutional features corresponding to any area image, the convolutional features can be input into each key point area feature layer, and each key point area feature layer extracts key point features respectively. If the corresponding human key point area is detected, the corresponding key point features are output. If the corresponding human key point area cannot be detected, the preset information is output. For example, for the upper left area image, after obtaining the convolutional features corresponding to the upper left area image, the convolutional features can be input into the head feature layer, the left / right arm feature layer, the torso feature layer, the left / right leg feature layer, and the left / right foot feature layer. Then, the head feature layer, the left arm feature layer, and the torso feature layer can detect the key points of the corresponding positions respectively, and output the key point features corresponding to the head area, the key point features corresponding to the left arm area, and the key point features corresponding to the torso area respectively. For the right arm feature layer, the left / right leg feature layer, and the left / right foot feature layer, the corresponding key point area cannot be detected, and a feature map of all 0 or 1 can be output.
[0161] like Figure 2 FIG. 1 is a schematic diagram of a key point feature extraction module of a second model provided in an embodiment of the present application, wherein Figure 2 In , image represents the pedestrian detection block diagram in any image, the four local features (localfeature) represent the upper left region image, the upper right region image, the lower left region image and the lower right region image respectively, and backbone(VIT)_new represents the trained first model. After the first model outputs the regional features of each regional image, the convolution layer (conv) of the second model can be used to convolve the regional features to obtain convolution features. Then, the convolution features are input to the head key point regional feature layer (head), the left arm / right arm key point regional feature layer (left / right arm), the torso key point regional feature layer (torso), the left leg / right leg key point regional feature layer (left / right leg), and the left foot / right foot key point regional feature layer (left / right foot) to obtain the key point features output by the key point regional feature layer. Figure 2 The different features in represent the key point features corresponding to different key point areas of the human body.
[0162] Exemplarily, for the upper left area image in the human body detection frame diagram, the key point features corresponding to the head area, the key point features corresponding to the left arm area, and the key point features corresponding to the torso area can be obtained. For the upper right area image in the human body detection frame diagram, the key point features corresponding to the head area, the key point features corresponding to the right arm area, and the key point features corresponding to the torso area can be obtained. For the lower left area image in the human body detection frame diagram, the key point features corresponding to the torso area, the key point features corresponding to the left leg area, and the key point features corresponding to the left foot area can be obtained. For the lower right area image in the human body detection frame diagram, the key point features corresponding to the torso area, the key point features corresponding to the right leg area, and the key point features corresponding to the right foot area can be obtained.
[0163] That is, each region image can obtain three key point features, and the three key point features corresponding to the upper left region image are Among them, when k is 0, 1, and 2, they correspond to the head area, left arm area, and torso area respectively. The three key point features corresponding to the image in the upper right area are Among them, when k is 0, 1, and 2, they correspond to the head area, right arm area, and torso area respectively. The three key point features corresponding to the image in the lower left area are Among them, when k is 0, 1, and 2, they correspond to the torso area, the left leg area, and the left foot area respectively. The three key point features corresponding to the image in the lower right area are Among them, when k is 0, 1, and 2, it corresponds to the torso area, the right leg area, and the right foot area respectively.
[0164] A2: Perform at least one convolution process on each key point feature and normalize the processed features to obtain the confidence level corresponding to each key point feature.
[0165] In the second model, each key point region feature layer may be sequentially connected to at least one convolution layer, and at least one convolution layer may be connected to a normalization layer (softmax). Figure 2 In FIG, a softmax layer is shown after each key point region feature layer, and a convolutional layer is not shown.
[0166] In specific implementation, after extracting the key point features from any key point region feature layer, the key point features can be sequentially input into the subsequent convolutional layer, and the key point features are first convolved at least once, and then the processed features are input into the softmax layer for normalization, so as to obtain the confidence corresponding to the key point features. Figure 2 In , the confidence of each key point feature is the confidence score.
[0167] For example, The corresponding confidence can be expressed as against The corresponding confidence can be expressed as against The corresponding confidence can be expressed as against The corresponding confidence can be expressed as
[0168] In one embodiment, S103 may be implemented according to B1 to B3 as follows:
[0169] B1: For each human key point region in each regional image, the key point features and confidence of the human key point region are weighted, and the weighted result and the key point feature are fused to obtain the first feature.
[0170] In specific implementation, for each key point region of the human body in each regional image, the feature fusion module in the second model can be used to first perform weighted processing on the key point features of the key point region of the human body and the confidence corresponding to the key point features to obtain a weighted result. Then, feature fusion is performed on the weighted result and the key point features to obtain the first feature with the camera style and pedestrian local identification information.
[0171] B2: Perform feature connection processing on the regional feature and the first feature to obtain the second feature.
[0172] In specific implementation, for any human key point region, the feature fusion module can be used to first perform convolution processing and normalization processing on the regional features of the regional image to which the human key point region belongs, so as to obtain the normalized regional features. Then, according to the principle of the residual network, the normalized regional features and the first features corresponding to the human key point region are subjected to feature connection (concat) processing to obtain the second features corresponding to the human key point. Among them, the feature connection processing can be, for example, in the form of a feature with the normalized regional features in front and the first features in the back, and the two are spliced into one feature.
[0173] B3: Use the convolutional attention module to fuse the second feature and the regional feature to obtain the fused features of the key point area of the human body.
[0174] Here, the Convolutional Block Attention Module (Local-attention map, CBAM for short) is a network module that focuses on spatial attention and channel attention.
[0175] In a specific implementation, for any key point area of the human body, the second feature corresponding to the key point area of the human body and the regional feature of the image of the region to which the key point area of the human body belongs can be input together into the CBAM module in the feature fusion module for feature fusion processing to obtain the fusion feature corresponding to the key point area of the human body. Here, through the feature fusion module, channel attention can be used to make the second model focus on the channel containing key pedestrian identification information. Through the combination of spatial attention and channel attention, double refinement of input features is achieved, and the second model can better focus on key features. Moreover, the model itself only uses simple global pooling and convolution operations, so that the model can maintain a low additional computing cost while bringing performance improvements.
[0176] For example, The corresponding fusion feature can be expressed as against The corresponding confidence can be expressed as against The corresponding confidence can be expressed as against The corresponding confidence can be expressed as
[0177] like Figure 3 FIG. 1 is a schematic diagram of a structure of a feature fusion module of a second model provided by an embodiment of the present disclosure. Figure 3 The feature in it represents the key point feature corresponding to any key point area of the human body, and the score represents the confidence score of the key point feature. Figure 3 The symbol 1 in represents weighted processing, Figure 3 The symbol 2 in represents feature fusion processing. Figure 3 The local feature in the figure represents the regional feature of the image of the region to which the key point region of the human body belongs. After weighting the confidence and key point features and fusing them with the key point features, the first feature is obtained. Conv and softmax represent convolution and normalization processing, concat represents feature connection processing, and the second feature is obtained after concat processing. CBAM represents convolution attention module, and Local-attention map represents the fusion feature corresponding to the key point region of the human body.
[0178] In one embodiment, the purpose of pedestrian re-identification is often to identify the pedestrian identification ID of each pedestrian in the image taken by the heterogeneous camera, which may include the pedestrian ID of the same pedestrian and the pedestrian ID of different pedestrians. Therefore, the pedestrian recognition result in the embodiment of the present disclosure may include the identification ID of the pedestrian. For the above S104, it can be implemented according to the following steps:
[0179] S104-1: Determine a reference camera and a matching camera among heterogeneous cameras.
[0180] Here, the heterogeneous cameras may include fisheye cameras and wide-angle cameras on the vehicle.
[0181] In a specific implementation, any one of the fisheye camera and the wide-angle camera can be used as a reference camera for pedestrian matching, and the other camera can be used as a matching camera for pedestrian matching.
[0182] Optionally, since the viewing angle of the fisheye camera is larger than that of the wide-angle camera, the pedestrians photographed by the fisheye camera may include all the pedestrians photographed by the wide-angle camera. Therefore, in a specific implementation, the wide-angle camera can be used as a reference camera and the fisheye camera as a matching camera, so that each pedestrian in the image taken by the fisheye camera can be matched once with each pedestrian in the image taken by the wide-angle camera.
[0183] S104-2: For any first pedestrian detection frame corresponding to the reference camera, determine the similarity between the first pedestrian in the first pedestrian detection frame and the second pedestrian in each second pedestrian detection frame according to the fusion features corresponding to each human key point area in the first pedestrian detection frame and the fusion features of each human key point area in each second pedestrian detection frame corresponding to the matching camera.
[0184] Here, the first pedestrian detection block diagram is the pedestrian detection block diagram in the image captured by the reference camera, and the first pedestrian is the pedestrian in the first pedestrian detection block diagram corresponding to the reference camera. The second pedestrian detection block diagram is the pedestrian detection block diagram in the image captured by the matching camera, and the second pedestrian is the pedestrian in the second pedestrian detection block diagram corresponding to the matching camera. One pedestrian detection block diagram includes one pedestrian.
[0185] Exemplarily, the feature matching module in the second model can be used to match the key points of the corresponding areas of the first pedestrian in any first pedestrian detection frame diagram in the image taken by the wide-angle camera with the fusion features of the key point areas of the human body corresponding to the first pedestrian in the first pedestrian detection frame diagram and the fusion features of the key point areas of the human body corresponding to the pedestrians in each second pedestrian detection frame diagram in the image taken by the matching camera, and the similarity between the first pedestrian and each second pedestrian can be determined based on the key point matching results.
[0186] For example, the first pedestrian may correspond to 12 fused features, and the second pedestrian may also correspond to 12 fused features. The 12 fused features of the first pedestrian may be matched with the 12 fused features of the second pedestrian according to the body parts corresponding to the key point regions of the human body and the regional images to which the key point regions of the human body belong, to obtain 12 key point matching results. For example, the head fused features of the upper left region image of the first pedestrian may be matched with the head fused features of the upper left region image of the second pedestrian, the left arm fused features of the upper left region image of the first pedestrian may be matched with the left arm fused features of the upper left region image of the second pedestrian, the torso fused features of the upper left region image of the first pedestrian may be matched with the torso fused features of the upper left region image of the second pedestrian, and so on, the right foot fused features of the lower right region of the first pedestrian may be matched with the right foot fused features of the lower right region of the second pedestrian, to obtain 12 key point matching results. Then, the similarity between the first pedestrian and the second pedestrian may be determined according to the 12 key point matching results.
[0187] For example, the 12 key point matching results may be 12 key point similarities, and then the pedestrian matching result between the first pedestrian and the second pedestrian is obtained by performing weighted summation on the 12 key point similarities.
[0188] Alternatively, for the first pedestrian or the second pedestrian, the fusion features corresponding to the pedestrian detection frame diagram to which the pedestrian belongs can be added and integrated to obtain the integrated features corresponding to the pedestrian. Then, the integrated features of the first pedestrian can be used to match the integrated features of each second pedestrian to obtain the pedestrian matching results between the first pedestrian and each second pedestrian.
[0189] S104-3: Determine the matching result between the first pedestrian and each second pedestrian according to the similarity; the matching result is used to indicate whether the second pedestrian is the same as the first pedestrian.
[0190] Here, the first pedestrian and the second pedestrian whose similarity is greater than a preset similarity can be regarded as the same pedestrian; and the first pedestrian and the second pedestrian whose similarity is not greater than the preset similarity can be regarded as different pedestrians.
[0191] Exemplarily, for any first pedestrian, the matching result between the first pedestrian and each second pedestrian may be determined according to the similarity between the first pedestrian and each second pedestrian and a preset similarity.
[0192] S104 - 4 : Determine an identification mark of each first pedestrian and an identification mark of each second pedestrian according to the matching result of the first pedestrian in each first pedestrian detection frame corresponding to the reference camera.
[0193] In a specific implementation, the same pedestrians and different pedestrians can be determined from the first pedestrians and the second pedestrians according to the matching results between each first pedestrian corresponding to the reference camera and each second pedestrian corresponding to the matching camera. Then, according to the preset identification generation rules, the identification identifier of each first pedestrian and the identification identifier of each second pedestrian are generated. The identification identifiers of the same first pedestrian and the second pedestrian are consistent, and the identification identifiers of different first pedestrians and the second pedestrian are different.
[0194] In this way, since the fusion features of each key point area of the human body are integrated with the local key point features and the global regional features, the influence of different internal and external parameters, resolutions, etc. of heterogeneous cameras on pedestrian recognition can be eliminated, thereby obtaining accurate pedestrian re-identification results.
[0195] In one embodiment, the vehicle-mounted heterogeneous camera in the embodiment of the present disclosure may include a wide-angle camera and a fisheye camera on the vehicle. Since the fisheye camera will cause more serious distortion, especially the distortion in the edge area of the image is more obvious, in order to eliminate the influence of the distortion of the fisheye camera on pedestrian re-identification, before executing S101, a first image including the pedestrian taken by the wide-angle camera and a second image including the pedestrian taken by the fisheye camera may also be obtained.
[0196] Here, since the image taken by the wide-angle camera has less distortion, the first image taken by the wide-angle camera can be directly used as the image for pedestrian re-identification. Since the second image taken by the fisheye camera has more serious distortion, the second image needs to be further processed to obtain an image that can be used for pedestrian re-identification.
[0197] Specifically, the second image may be dedistorted to obtain a third image corresponding to the fisheye camera. Exemplarily, a dedistortion algorithm may be used to dedistort the second image taken by the fisheye camera to obtain a third image corresponding to the fisheye camera, and the third image may be used as an image for pedestrian re-identification.
[0198] Optionally, a dedistortion algorithm may be used to perform dedistortion processing on the image taken by the wide-angle camera to obtain a dedistorted image corresponding to the wide-angle camera.
[0199] Furthermore, with respect to the step of "performing pedestrian detection on the image, and cropping the image according to the pedestrian detection frame to obtain multiple pedestrian detection frame diagrams" in S101, pedestrian detection can be performed on the first image and the third image respectively, and the first image and the third image can be cropped respectively according to the pedestrian detection frame to obtain multiple pedestrian detection frame diagrams.
[0200] Exemplarily, the pedestrian detection model can be used to perform pedestrian detection on the first image taken by the wide-angle camera and the third image corresponding to the fisheye camera, respectively, to obtain each pedestrian detection frame in the first image taken by the wide-angle camera and each pedestrian detection frame in the third image corresponding to the fisheye camera. Then, the first image can be cropped using each pedestrian detection frame in the first image to obtain each pedestrian detection frame diagram corresponding to the first image, and the third image can be cropped using each pedestrian detection frame in the third image to obtain each pedestrian detection frame diagram corresponding to the third image.
[0201] Based on the above embodiments, it can be seen that the pedestrian re-identification method provided by the embodiment of the present disclosure needs to use the first model and the second model, so the embodiment of the present disclosure also provides a method for training the first model and the second model. The training process of the first model and the second model will be described separately below:
[0202] For the first model, it can be trained according to the following steps:
[0203] Step 1: Obtain a sample image dataset; the sample image dataset includes a plurality of sample image pairs, and the sample image pairs include sample images respectively taken by heterogeneous cameras.
[0204] Here, images including pedestrians taken by heterogeneous cameras can be collected as sample images, and then the sample images taken by the heterogeneous cameras at the same time can be taken as a sample image pair, and each sample image pair is used to form a sample image dataset. It should be noted that each sample image can be an image processed using a de-distortion algorithm.
[0205] When it is determined that the first model needs to be trained, a sample image dataset may be obtained.
[0206] Step 2: For any sample image pair, the sample pedestrian detection frame diagram in each sample image included in the sample image pair is input into the first model to be trained to obtain the predicted area features corresponding to the multiple sample area images in the sample pedestrian detection frame diagram.
[0207] Here, the sample pedestrian detection frame diagram is the pedestrian detection frame diagram in the sample image, and the sample region image is each region image in the sample pedestrian detection frame diagram.
[0208] In specific implementation, for any sample image pair in the sample image data set, the pedestrian detection model can be used to detect the sample pedestrian detection frame corresponding to each sample image in the sample image pair and perform image cropping to obtain the sample pedestrian detection frame diagram in each sample image. Then, the sample pedestrian detection frame diagram in the sample image can be input into the first model to be trained. The first model will first divide the sample pedestrian detection frame diagram into regions to obtain four sample region images corresponding to the sample pedestrian detection frame diagram, and randomly intercept the sample target sub-region image from each sample region image according to the interception ratio. Then, for each sample region image, the sample target sub-region images corresponding to the four sample region images can be simultaneously input into the encoding module (encoder), and the encoder is used to extract features from each sample target sub-region image to obtain the predicted region features corresponding to each sample target sub-region image. In this way, the predicted region features corresponding to the sample target sub-region images of each sample region image constitute the predicted region features corresponding to multiple sample region images in the sample pedestrian detection frame diagram.
[0209] Step three: merge the predicted region features corresponding to the sample region image and the manually constructed shared features, and input the merged features into the decoding module in the first model to be trained to obtain the restored region image corresponding to the sample region image.
[0210] Here, the shared feature is an artificially constructed feature, but it is also a learnable feature. Based on the above embodiments, it can be seen that when the first model extracts features from the regional image, it needs to discard 75% of the image area (that is, it needs to mask 75% of the local views), so the shared features here can be used to characterize the features corresponding to the discarded (that is, masked) parts of the sample regional image. The shared feature can be represented as a masked token. It is understandable that the shared features corresponding to each sample regional image in each sample image are the same.
[0211] In a specific implementation, for any sample region image, after obtaining the predicted region feature corresponding to the sample region image, the predicted region feature and the shared feature can be merged to obtain a merged feature. Then, the merged feature is input into the decoding module (decoder) in the first model to be trained to obtain the restored region image corresponding to the sample region image. For example, the predicted region feature and the shared feature corresponding to the upper left region image of the sample image are merged to obtain a merged feature. The merged feature is then input into the decoder to obtain the restored region image corresponding to the upper left region image.
[0212] Step 4: Determine the first loss of the first model to be trained according to the sample region images corresponding to the sample image pairs and the restored region image corresponding to each sample region image.
[0213] Here, the first loss is the loss of the first model when performing feature extraction, which is a loss similar to the mean absolute error (MSE).
[0214] In a specific implementation, the first loss of the first model may be determined according to an error between a sample region image corresponding to each sample image in each sample image pair and a restored region image corresponding to the sample region image.
[0215] Step 5: Iteratively train the first model to be trained using the first loss until the training cutoff condition is met to obtain the trained first model.
[0216] Here, the training cutoff condition may be that the number of iterative training rounds reaches a preset number of rounds and / or the prediction accuracy of the trained network model reaches a preset accuracy.
[0217] In a specific implementation, after constructing the first loss, the first model to be trained can be iteratively trained using the first loss, and when a training cutoff condition is met, the trained first model is used as the trained first model.
[0218] In this way, self-supervised training is first performed on unlabeled data of heterogeneous cameras with different styles, and then the pre-trained backbone (VIT) is used for training with labeled data, which can make the model adapt to the style transformation of heterogeneous cameras and increase the performance of feature extraction under heterogeneous cameras. In addition, using images taken by heterogeneous cameras as sample images and using methods similar to MAE, the encoder will focus on the target information in images of different styles rather than redundant information on the one hand, and the model can extract richer image details from only a small range of images on the other hand, which will significantly increase the model's data feature extraction capabilities under different image styles of heterogeneous cameras and focus on feature extraction in the four overlapping areas set for the pedestrian detection frame diagram.
[0219] It should be noted that after the first model is trained, when performing person re-identification, only the encoder in the trained first model can be used to extract regional features from the regional image, and the decoder in the first model will no longer be used to restore the regional image. That is, the decoder is only used in the process of training the first model, and will not be used in the process of applying the first model.
[0220] like Figure 4As shown, it is a structural schematic diagram of a first model provided by an embodiment of the present disclosure, wherein: Figure 4 The image in the sample image represents the sample pedestrian detection frame diagram in the sample image, local(up1) represents the sample region image corresponding to the upper left region of the sample pedestrian detection frame diagram, and local(up2) represents the sample region image corresponding to the upper right region of the sample pedestrian detection frame diagram; local(down1) represents the sample region image corresponding to the lower left region of the sample pedestrian detection frame diagram, and local(down2) represents the sample region image corresponding to the lower right region of the sample pedestrian detection frame diagram. Encoder is the encoder in the first model built using backbone(VIT) as the framework. Figure 4 The solid boxes output by the Encoder in represent the predicted regional features corresponding to different sample region images. Figure 4 The dashed boxes in represent shared features. Decoder is the decoder in the first model built using backbone (VIT) as the framework. Figure 4 Each black box output by the decoder in represents the restored region image corresponding to the different sample region images output by the decoder, and each restored region image can constitute the restored sample image. During training, the sample region image and the restored region image corresponding to the sample region image output by the decoder can be used to construct the first loss.
[0221] In another embodiment, the second model can be trained according to the following steps:
[0222] S1: Obtain sample area features of each sample area image in multiple sample pedestrian detection frame diagrams; the sample area features are obtained by extracting features of the sample pedestrian detection frame diagram in the sample image using the trained first model; the sample image is an image in a sample image pair; the sample image pair includes sample images taken separately using heterogeneous cameras.
[0223] In a specific implementation, for any sample image in the sample image pair, the trained first model can be used to perform feature extraction processing on the sample image to obtain sample region features of each sample region image output by the encoder in the trained first model.
[0224] S2: Input the sample area features into the second model to be trained to extract human key points, determine the predicted key point features of each human key point area corresponding to each sample area image in each sample pedestrian detection frame diagram and the prediction confidence corresponding to the predicted key point features.
[0225] Here, the predicted key point feature is the feature of the human key point region in the sample region image, and the prediction confidence is the confidence of the predicted key point feature.
[0226] In specific implementation, the sample region features can be input into the second model to be trained. The key point feature extraction module in the second model to be trained is used to perform key point detection on the sample region image to detect each human key point region in the sample region image. For each human key point region detected, the predicted key point features of each human key point region and the prediction confidence corresponding to each predicted key point feature can be extracted.
[0227] S3: Determine the prediction category of each pixel point in each sample area image by using the prediction key point features and prediction confidence.
[0228] In specific implementation, the key point feature extraction module in the second model to be trained can be used to determine the pixel points corresponding to each category of the human key point area in the sample area image according to each predicted key point feature and prediction confidence in the sample area image, so that the predicted category of each pixel point in the sample area image can be obtained. The predicted category is used to indicate the body part category to which the pixel point belongs, and the body part category can include the head category, left arm category, right arm category, torso category, left leg category, right leg category, left foot category, right foot category and empty category, wherein the empty category means that the pixel point does not belong to the human key point corresponding to any body part.
[0229] Taking the upper left sample area image in the sample image as an example, the predicted key point features and predicted confidence corresponding to the head area in the upper left sample area image can be weighted, and the pixel points corresponding to the head area in the upper left sample area image can be determined according to the weighted result. For example, when the weighted result is greater than the preset value, the category of each pixel point in the head area is determined as the head category, otherwise, it is determined as an empty category. Similarly, the pixel points corresponding to the left arm area in the upper left sample area image, the pixel points corresponding to the torso area in the upper left sample area image, and the pixel points that neither belong to the head area nor the left arm area and the torso area can be obtained. In this way, the predicted category of each pixel point in the upper left sample area image can be obtained.
[0230] S4: feature fusion is performed on the predicted key point features, the predicted confidence and the sample area features to obtain the predicted fusion features corresponding to each human key point area in each sample area image in each sample pedestrian detection frame diagram, and the predicted identification of the pedestrian in each sample pedestrian detection frame diagram is determined based on the predicted fusion features.
[0231] Here, the predicted fusion feature is the fusion feature corresponding to the human key point area in the sample area image, and the predicted identifier is the pedestrian identifier of the pedestrian in the sample pedestrian detection frame diagram.
[0232] During specific implementation, the specific implementation process of the step of "using predicted key point features, predicted confidence and sample area features to perform feature fusion to obtain predicted fusion features corresponding to each human key point area" can be performed using the feature fusion module in the second model to be trained. For specific execution steps, please refer to the introduction of S103 above, which will not be repeated here. For the step of "determining the predicted identification of pedestrians in each sample pedestrian detection frame diagram based on the predicted fusion features", the feature matching module in the second model to be trained can be performed. For specific execution steps, please refer to the introduction of S104 above, which will not be repeated here.
[0233] S5: Determine the second loss of the second model to be trained according to the predicted category and label category of each pixel point in the sample pedestrian detection frame diagram, the predicted identifier and label identifier of the pedestrian in the sample pedestrian detection frame diagram, and the predicted fusion feature, and use the second loss to iteratively train the second model to be trained until the training cutoff condition is met to obtain a trained second model.
[0234] Here, the second loss is the prediction loss of the second model, which can be divided into two parts. One part is the loss of the key point feature extraction module when predicting the key point category, and the other part is the loss of the feature fusion module and the feature matching module when predicting pedestrian identification.
[0235] The label category is the real category of each pixel in the sample pedestrian detection frame map. The label identifier is used to indicate the real identifier of the pedestrian in the sample pedestrian detection frame map.
[0236] Therefore, in specific implementation, the prediction category of each pixel point in each sample pedestrian detection frame diagram included in each sample image and the label category of each pixel point can be used to determine the prediction loss of the key point feature extraction module; at the same time, the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram and the prediction fusion feature can be used to determine the loss when predicting the pedestrian identification, and the two parts of the loss are summed to obtain the second loss. After obtaining the second loss, the second loss can be used to iteratively train the second model as a whole, and when the training cutoff condition is met, the obtained second model is used as the trained second model.
[0237] It is understandable that, for the key point feature extraction module, in the process of training the second model, the key point feature extraction module can be used to output the predicted category of each pixel in each sample area image. However, when applying the trained second model, only the key point feature extraction module can be used to output the key point features of each human key point area in the area image and the confidence corresponding to the key point features, without the need to output the category of each pixel in the area image.
[0238] In one embodiment, the step of determining the second loss in S5 may be implemented as follows:
[0239] S5-1: Determine the key point prediction loss based on the prediction category and label category of each pixel point in the sample pedestrian detection frame map.
[0240] Here, the predicted category of each pixel in the sample pedestrian detection frame is the predicted category of each pixel included in each sample area image in the sample pedestrian detection frame. The key point prediction loss is the loss of the key point feature extraction module in the second model to be trained when predicting the key point category.
[0241] In specific implementation, the predicted category and label category of each pixel in the sample pedestrian detection frame diagram can be used to calculate the cross entropy loss, and the obtained cross entropy loss can be used as the key point prediction loss of the second model to be trained.
[0242] Exemplarily, the key point prediction loss may be determined according to the following formula 1:
[0243]
[0244] Among them, L att represents the key point prediction loss, x represents the x-th pixel of the sample pedestrian detection frame map, X is the total number of pixels in the sample pedestrian detection frame map, H represents the height of the sample pedestrian detection frame map, W represents the width of the sample pedestrian detection frame map, i, j represent the horizontal and vertical coordinates of the pixel in the sample pedestrian detection frame map, respectively, S ij Indicates the predicted category corresponding to the pixel point with coordinates (i, j), y ij Indicates the label category corresponding to the pixel with coordinates (i, j).
[0245] In the specific implementation, the prediction category and label category of each pixel point in each sample pedestrian detection frame map are substituted into the above formula 1, and the key point prediction loss L of the second model to be trained can be obtained. att .
[0246] S5-2: Determine the recognition loss based on the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram, and the predicted fusion features.
[0247] Here, the recognition loss is the prediction loss of the feature fusion module and the feature matching module in the second model to be trained when performing pedestrian identification prediction.
[0248] In specific implementation, the recognition loss of the second model to be trained can be determined according to the following steps:
[0249] S5-2-1: Determine the identification prediction loss based on the predicted identification of the pedestrians in each sample pedestrian detection frame diagram, the label identification of each pedestrian, and the first number of pedestrians.
[0250] Here, the first number is the number of sample pedestrian detection frame diagrams used in the process of training the second model. Since one sample pedestrian detection frame diagram includes one pedestrian, this number is also the number of pedestrians corresponding to the sample pedestrian detection frame diagram.
[0251] Optionally, in order to reduce the amount of calculation, the first number may also be the number of sample pedestrian detection block diagrams corresponding to a camera in the heterogeneous cameras in the sample pedestrian detection block diagram used when training the second model, for example, the number of sample pedestrian detection block diagrams corresponding to a wide-angle camera.
[0252] The identification prediction loss is used to characterize the prediction loss of the feature matching model when predicting pedestrian identification.
[0253] In specific implementation, the predicted identification of the pedestrian, the label identification of the pedestrian, and the first number of pedestrians can be used to determine the identification prediction loss using the label smoothing cross entropy loss function. For example, the identification prediction loss can be determined using the following formula 2 and formula 3:
[0254]
[0255]
[0256] Among them, L id represents the identification prediction loss, N represents the first number of pedestrians, different values of h represent different pedestrians, and p h represents the predicted identifier corresponding to the h-th pedestrian, y represents the label identifier, that is, the ground-truth ID label of the pedestrian, and ε is a preset smoothing parameter, which can be set to 0.1, for example.
[0257] Exemplarily, after the predicted identification of the pedestrians in each sample pedestrian detection block diagram is outputted by the second model to be trained, the predicted identification of the pedestrians, the label identification of the pedestrians, and the first number of pedestrians can be input into the above formulas 2 and 3 to obtain the identification prediction loss L of the second model to be trained. id .
[0258] S5-2-2: For any first sample pedestrian detection frame diagram in each sample pedestrian detection frame diagram, determine the positive sample set and the negative sample set corresponding to the first sample pedestrian detection frame diagram from the second sample pedestrian detection frame diagram based on the first label identification of the pedestrian in the first sample pedestrian detection frame diagram and the second label identification of the pedestrian in each second sample pedestrian detection frame diagram except the first sample pedestrian detection frame diagram.
[0259] Here, the first sample pedestrian detection block diagram may be any sample pedestrian detection block diagram among the sample pedestrian detection block diagrams, and the second sample pedestrian detection block diagram may be each sample pedestrian detection block diagram among the sample pedestrian detection block diagrams except the first sample pedestrian detection block diagram.
[0260] The first label identification is the label identification of the pedestrian in the first sample pedestrian detection frame diagram, and the second label identification is the label identification of the pedestrian in the second sample pedestrian detection frame diagram.
[0261] The positive sample set includes each second sample pedestrian detection frame diagram whose corresponding second label identifier is the same as the first label identifier, and the negative sample set includes each second sample pedestrian detection frame diagram whose corresponding second label identifier is different from the first label identifier.
[0262] In a specific implementation, for each sample pedestrian detection frame graph used when training the second model to be trained, any one of the sample pedestrian detection frame graphs can be selected as the first sample pedestrian detection frame graph, and the other sample pedestrian detection frame graphs can be selected as the second sample pedestrian detection frame graph. Then, from the second sample pedestrian detection frame graph, each second sample pedestrian detection frame graph whose second label identification of the pedestrian is consistent with the first label identification of the pedestrian in the first sample pedestrian detection frame graph is screened out to form a positive sample set, and from the second sample pedestrian detection frame graph, each second sample pedestrian detection frame graph whose second label identification of the pedestrian is inconsistent with the first label identification of the pedestrian in the first sample pedestrian detection frame graph is screened out to form a negative sample set.
[0263] In one embodiment, the above S5-2-2 may also be implemented according to the following steps:
[0264] S5-2-2-1: Select a second number of target pedestrians from the first number of pedestrians.
[0265] Here, the second number is smaller than the first number, and the second number can be set according to the number of sample pedestrian detection frames used.
[0266] Exemplarily, the first number is N, the second number is M, and when selecting the positive and negative sample sets, M target pedestrians can be selected from the N pedestrians.
[0267] S5-2-2-2: For each target pedestrian, based on the third label identification and each second label identification of the target pedestrian, filter out a third number of third sample pedestrian detection frame diagrams corresponding to each target pedestrian from each second sample pedestrian detection frame diagram.
[0268] Here, the second number and the third number can be preset according to the batch size (batch_size) required for training the second model. For example, the second number is M, and the third number can be J. The third label identifier is the real label identifier of the target pedestrian.
[0269] In a specific implementation, for each target pedestrian, the third label identifier and each second label identifier of each target pedestrian can be used to randomly select a third sample pedestrian detection frame whose second label identifier is consistent with the third label identifier from each second sample pedestrian detection frame diagram, wherein the number of the third sample pedestrian detection frames is the third number. That is, the third sample pedestrian detection frame diagram corresponding to any target pedestrian is the sample pedestrian detection frame diagram including the target pedestrian.
[0270] Exemplarily, for each target pedestrian, a third number of third sample pedestrian detection frames whose label identification is consistent with the target pedestrian can be randomly selected from the second sample pedestrian detection frame map. For example, the second number is M and the third number can be J. In this case, a batch includes M×J sample pedestrian detection frame maps.
[0271] S5-2-2-3: Based on the first label identification and the third label identification of each target pedestrian, determine from the third sample pedestrian detection frame diagram corresponding to each target pedestrian, a set of positive samples with the same identification as that corresponding to the first sample pedestrian detection frame diagram, and a set of negative samples with different identifications corresponding to the first sample pedestrian detection frame diagram.
[0272] Exemplarily, the first label identification of the pedestrian in the first sample pedestrian detection frame diagram and the third label identification of each target pedestrian can be used to take each third sample pedestrian detection frame diagram corresponding to the target pedestrian whose third label identification is consistent with the first label identification as the positive sample set corresponding to the first sample pedestrian detection frame diagram with the same identification; and take each third sample pedestrian detection frame diagram corresponding to the target pedestrian whose third label identification is inconsistent with the first label identification as the negative sample set corresponding to the first sample pedestrian detection frame diagram with different identifications. That is, the positive sample set includes each third sample pedestrian detection frame diagram whose label identification is consistent with the first label identification, and the negative sample set includes each third sample pedestrian detection frame diagram whose label identification is inconsistent with the first label identification.
[0273] S5-2-3: Determine the triplet loss by using the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the positive sample set, and the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the negative sample set.
[0274] For the sake of ease of description, the second sample pedestrian detection frame diagram (or the third sample pedestrian detection frame diagram) in the positive sample set is called the positive sample of the first sample pedestrian detection frame diagram, and the second sample pedestrian detection frame diagram (or the third sample pedestrian detection frame diagram) in the negative sample set is called the negative sample of the first sample pedestrian detection frame diagram.
[0275] In specific implementation, the first Euclidean distance between the first sample pedestrian detection frame diagram and each positive sample can be determined based on the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram and the predicted fusion features of each human key point area corresponding to each positive sample in the positive sample set; and the second Euclidean distance between the first sample pedestrian detection frame diagram and each negative sample can be determined based on the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram and the predicted fusion features of each human key point area corresponding to each negative sample in the negative sample set.
[0276] Then, the triple loss can be constructed with the goal of maximizing the first Euclidean distance and minimizing the second Euclidean distance. Alternatively, the largest first Euclidean distance can be selected from the first Euclidean distances, and the smallest second Euclidean distance can be selected from the second Euclidean distances, and the triple loss can be constructed using the largest first Euclidean distance and the smallest second Euclidean distance.
[0277] Optionally, in order to reduce the amount of calculation, when calculating the first Euclidean distance between the first sample pedestrian detection block diagram and the positive sample, multiple target prediction fusion features can be randomly selected from the prediction fusion features of each human key point area corresponding to the first sample pedestrian detection block diagram. Among them, the number of the multiple target prediction fusion features selected is less than the number of prediction fusion features of each human key point area corresponding to the first sample pedestrian detection block diagram. Based on the above embodiment, it can be known that the corresponding prediction fusion features in a sample pedestrian detection block diagram can be 12, and 8 prediction fusion features can be randomly selected here as target prediction fusion features. Then, the first Euclidean distance between the first sample pedestrian detection block diagram and the positive sample can be calculated using the multiple target prediction fusion features corresponding to the first sample pedestrian detection block diagram and the prediction fusion features in the positive sample that match the target prediction fusion features. Similarly, the second Euclidean distance can also be calculated using multiple randomly selected target prediction fusion features, which will not be repeated here.
[0278] In one embodiment, the above S5-2-3 can be implemented according to the following steps:
[0279] S5-2-3-1: For any second sample pedestrian detection frame diagram in the positive sample set, select multiple first target features from the predicted fusion features of each human body key point area corresponding to the second sample pedestrian detection frame diagram.
[0280] Here, for any sample pedestrian detection block diagram, the predicted fusion features of the human key point area output by the feature fusion model can include 12, namely and Among them, k = {0, 1, 2}.
[0281] It can be understood that when the positive sample set includes positive samples selected from the third sample pedestrian detection block diagram corresponding to the target pedestrian, the second sample pedestrian detection block diagram used in S5-2-3-1 can be the third sample pedestrian detection block diagram belonging to the positive sample.
[0282] Therefore, in a specific implementation, for any second sample pedestrian detection block diagram in the positive sample set, multiple predicted fusion features can be randomly selected from the 12 predicted fusion features corresponding to the second sample pedestrian detection block diagram as the first target features. For example, 8 first target features can be selected from the 12 predicted fusion features.
[0283] Optionally, in order to further reduce the amount of calculation, a positive sample that is most difficult to match with the first sample pedestrian detection frame diagram may be selected from the positive samples included in the positive sample set, and multiple first target features may be selected from the predicted fusion features of each human key point region corresponding to the positive sample. Among them, the positive sample that is most difficult to match can be understood as a positive sample that the pedestrian in the second sample pedestrian detection frame diagram looks the least similar to the pedestrian in the first sample pedestrian detection frame diagram, but belongs to the same pedestrian.
[0284] S5-2-3-2: for any second sample pedestrian detection frame diagram in the negative sample set, select multiple second target features from the predicted fusion features of each human key point area corresponding to the second sample pedestrian detection frame diagram;
[0285] It can be understood that when the negative sample set includes negative samples selected from the third sample pedestrian detection block diagram corresponding to the target pedestrian, the third sample pedestrian detection block diagram used in S5-2-3-2 can be the third sample pedestrian detection block diagram belonging to the negative sample.
[0286] In a specific implementation, for any second sample pedestrian detection block diagram in the negative sample set, multiple predicted fusion features can be randomly selected from the 12 predicted fusion features corresponding to the second sample pedestrian detection block diagram as the second target features. For example, 8 second target features can be selected from the 12 predicted fusion features.
[0287] Optionally, in order to further reduce the amount of calculation, the negative sample that is most difficult to match with the first sample pedestrian detection frame diagram may be selected from the negative samples included in the negative sample set, and multiple second target features may be selected from the predicted fusion features of each human key point area corresponding to the negative sample. Among them, the negative sample with the greatest matching difficulty may be understood as a negative sample in which the pedestrian in the second sample pedestrian detection frame diagram looks most similar to the pedestrian in the first sample pedestrian detection frame diagram, but belongs to different pedestrians.
[0288] S5-2-3-3: From the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, select the third target feature that matches each first target feature and the fourth target feature that matches each second target feature.
[0289] Here, the third target feature that matches the first target feature is the predicted fusion feature corresponding to the same human key point region as the first target feature among the 12 predicted fusion features corresponding to the first sample pedestrian detection frame diagram. For example, when the first target feature is the predicted fusion feature corresponding to the head key point region, the third target feature is the predicted fusion feature corresponding to the head key point region among the 12 predicted fusion features corresponding to the first sample pedestrian detection frame diagram.
[0290] The fourth target feature is the predicted fusion feature corresponding to the same human key point region as the second target feature among the 12 predicted fusion features corresponding to the first sample pedestrian detection frame diagram. For example, if the second target feature is the predicted fusion feature corresponding to the left foot key point region, the fourth target feature is the predicted fusion feature corresponding to the left foot key point region among the 12 predicted fusion features corresponding to the first sample pedestrian detection frame diagram.
[0291] For example, after selecting 8 first target features from the positive sample, 8 third target features corresponding to the same body part can be selected from the first sample pedestrian detection frame diagram. And, after selecting 8 second target features from the negative sample, 8 fourth target features corresponding to the same body part can be selected from the first sample pedestrian detection frame diagram.
[0292] S5-2-3-4: Determine the triplet loss based on the Euclidean distance between each first target feature and the matching third target feature, and the Euclidean distance between each second target feature and the matching fourth target feature.
[0293] In a specific implementation, the first Euclidean distance between the first sample pedestrian detection frame diagram and the positive sample can be determined according to the Euclidean distance between each first target feature and the matching third target feature; and the second Euclidean distance between the first sample pedestrian detection frame diagram and the negative sample can be determined according to the Euclidean distance between each second target feature and the matching fourth target feature. Then, the triplet loss can be determined using the first Euclidean distance, the second Euclidean distance and the batch_size.
[0294] Exemplarily, the triplet loss may be determined according to the following formula 4 and formula 5:
[0295]
[0296] d a,b =‖f a -f b ‖2; (Formula 5)
[0297] Among them, L TriHard represents the triplet loss of the second model to be trained; M×J represents a batch of M×J samples of pedestrian detection block diagram; 8 represents 8 predicted fusion features randomly selected from 12 predicted fusion features. These 8 predicted fusion features are the first target features for positive samples and the second target features for negative samples. Set A is the positive sample set, set B is the negative sample set, p represents the first label identifier, p + represents a positive sample, n represents a negative sample, and a represents the first sample pedestrian detection block diagram; represents the Euclidean distance between the first sample pedestrian detection frame and the positive sample; d a,n represents the Euclidean distance between the first sample pedestrian detection frame and the negative sample; α is a preset parameter; d a,b Represents the vector f a and vector f b The Euclidean distance between a It can represent the third target feature or the fourth target feature corresponding to the first sample pedestrian detection frame diagram; f b It can represent multiple first target features corresponding to positive samples, or it can represent multiple second target features corresponding to negative samples. The plus sign in If the value of is greater than or equal to 0, the value itself is returned; If the value is less than 0, it returns 0.
[0298] In specific implementation, each first target feature and the matching third target feature, as well as each second target feature and the matching fourth target feature, can be substituted into the above formula 4 and formula 5 to obtain the triplet loss. In this way, through the TriHardloss at the body part level (i.e., key point level), the model pays more attention to the invariant identification features between cross-camera images, and at the same time, combined with the identification prediction loss, the model can also identify the target identity from a global perspective.
[0299] S5-2-4: Determine the recognition loss based on the identification prediction loss and the triplet loss.
[0300] In specific implementation, the recognition loss can be determined according to the following formula 6:
[0301] L Reid =L id +L TriHard ; (Formula 6)
[0302] Among them, L Reed represents the recognition loss of the second model to be trained, L id represents the identification prediction loss of the second model to be trained, L TriHard Represents the triplet loss of the second model to be trained.
[0303] That is, the above formula 6 can be used to add the identification prediction loss and the triple loss to obtain the recognition loss.
[0304] S5-3: Determine the second loss based on the key point prediction loss and the identification loss.
[0305] In specific implementation, the second loss can be determined according to the following formula 7:
[0306] L=LReid +λL att ; Formula 7
[0307] Among them, L represents the second loss, L Reid represents the recognition loss of the second model to be trained, and λ is a preset adjustment parameter used to adjust the contribution of the loss corresponding to the key point feature extraction module.
[0308] Exemplarily, the key point prediction loss and the preset adjustment parameter can be multiplied, and the multiplication result and the recognition loss can be added to obtain the second loss.
[0309] In this way, the pre-trained backbone is already capable of learning data features under different viewing angles of heterogeneous cameras. At the same time, by using the key point feature extraction module, the model can pay more attention to the common features of the same ID between heterogeneous cameras and actively punish occlusion information based on the score. The feature fusion module enables the model to learn and distinguish global features and local similar features from multiple scales, avoiding interference from local similar information.
[0310] like Figure 5 FIG. 1 is a schematic diagram of a second loss determination process provided by an embodiment of the present disclosure, wherein: Figure 5 The leftmost image in the figure represents the sample pedestrian detection frame diagram. Figure 5 In which local(up1) represents the sample area image corresponding to the upper left area of the sample pedestrian detection frame diagram, local(up2) represents the sample area image corresponding to the upper right area of the sample pedestrian detection frame diagram; local(down1) represents the sample area image corresponding to the lower left area of the sample pedestrian detection frame diagram, and local(down2) represents the sample area image corresponding to the lower right area of the sample pedestrian detection frame diagram. Figure 5 The backbone(VIT)_new in represents the encoder in the trained first model. Figure 5 The four key point feature extraction modules in are used to extract the predicted key point features and prediction confidences corresponding to the four sample area images respectively, and use the predicted key point features and prediction confidences to determine the predicted category of each pixel in the sample area image. Then, the key point prediction loss is determined based on the predicted category and label category. The predicted key point features and prediction confidences are input into the feature fusion module, and the predicted fusion features corresponding to each human key point area are output. The predicted identification of the pedestrian is determined based on the predicted fusion features. Then, the identification prediction loss (ID loss) and the triplet loss (i.e., TriHardloss) can be determined using the predicted identification and label identification.
[0311] Based on the above embodiments, the present disclosure designs a training scheme that takes into account both internal and external parameter transformations and different resolutions, and proposes two modules. Among them, in order to solve the problem of image style transformation caused by different internal parameters of heterogeneous cameras, an unsupervised pre-training method is used to crop pedestrians in the pedestrian image dataset collected by wide-angle and fisheye cameras (i.e., the sample image dataset mentioned above) and then pre-train them. In this way, a backbone that can focus on data under heterogeneous camera styles is pre-trained. Compared with the model pre-trained on the ImageNet dataset, pre-training on heterogeneous camera datasets can make the model focus on the Re-ID task.
[0312] To address the problem of poor camera matching accuracy caused by lens distortion due to different intrinsic parameters of heterogeneous cameras, a key point feature extraction module is proposed. By using human key point labels (i.e., the label categories mentioned above) for supervised learning, the key point feature extraction module can extract body part features, which is conducive to the key point feature extraction module focusing on the invariant features in heterogeneous cameras. Even in the case of distortion, the features at the key points of the human body are invariant or have extremely similar characteristics. At the same time, in order to address the occlusion problem caused by changes in the external parameters of heterogeneous cameras, a confidence score is added to determine whether the extracted features are features under occlusion.
[0313] A feature fusion module is designed to solve the problem of similar local features caused by different resolutions of heterogeneous cameras. It allows the extracted body part features to be fused in multi-level features (i.e., local and global fusion) to enhance the learning and representation of occlusion and similar features. By fusing body part features (i.e., key point features) and global features (i.e., regional features), the model not only learns the local features of the body parts, but also learns the ID discriminant features that may exist in non-body parts.
[0314] In general, the disclosed embodiments propose solutions to the difficult problems of heterogeneous camera Re-ID (such as different camera viewing angles, occlusion problems, and local feature similarity problems), so that better results can be achieved in pedestrian re-identification tasks in heterogeneous camera style change scenarios. Solutions are proposed for the distortion problems and viewing angle problems caused by different camera positions and principles, and identity recognition is performed when pedestrians are occluded in heterogeneous cameras. Similar targets under heterogeneous camera lenses can be more accurately identified, greatly improving the accuracy of pedestrian re-identification in heterogeneous camera scenarios.
[0315] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0316] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0317] Based on the same technical concept, the embodiment of the present application also provides a computer device. Figure 6 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application, including:
[0318] Processor 601 , memory 602 and bus 603 . Among them, the memory 602 stores machine-readable instructions executable by the processor 601, and the processor 601 is used to execute the machine-readable instructions stored in the memory 602. When the machine-readable instructions are executed by the processor 601, the processor 601 performs the following steps: S101: obtaining images including pedestrians collected by the vehicle-mounted heterogeneous cameras, and performing pedestrian detection on the images, and cropping the images according to the pedestrian detection frames to obtain multiple pedestrian detection frame diagrams; S102: using a pre-trained first model to determine the image features corresponding to each pedestrian detection frame diagram, and input the image features into a pre-trained second model to extract human key points, and determine the key point features of each human key point area in each pedestrian detection frame diagram and the confidence corresponding to the key point features; S103: performing feature fusion on the key point features, confidence and image features to obtain the fusion features corresponding to each human key point area in each pedestrian detection frame diagram; and S104: determining the pedestrian recognition result corresponding to each pedestrian detection frame diagram according to the fusion features of each human key point area in the pedestrian detection frame diagram corresponding to the heterogeneous cameras.
[0319] The above-mentioned memory 602 includes a memory 6021 and an external memory 6022; the memory 6021 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 601, as well as the data exchanged with the external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the memory 6021. When the computer device is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the execution instructions mentioned in the above method embodiment.
[0320] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the pedestrian re-identification method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0321] The computer program product of the pedestrian re-identification method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the pedestrian re-identification method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0322] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.
[0323] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0324] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0325] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0326] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0327] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0328] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A pedestrian re-identification method, characterized in that: include: Acquire images including pedestrians respectively captured by vehicle-mounted heterogeneous cameras, perform pedestrian detection on the images, and crop the images according to pedestrian detection frames to obtain multiple pedestrian detection frame diagrams; Using the pre-trained first model, determining the image features corresponding to each of the pedestrian detection block diagrams, and inputting the image features into the pre-trained second model to extract human key points, and determining the key point features of each human key point area in each of the pedestrian detection block diagrams and the confidence corresponding to the key point features; Performing feature fusion on the key point features, the confidence level and the image features to obtain fusion features corresponding to each key point area of the human body in each pedestrian detection frame diagram; According to the fusion features of each of the human body key point areas in the pedestrian detection block diagrams respectively corresponding to the heterogeneous cameras, a pedestrian recognition result corresponding to each of the pedestrian detection block diagrams is determined.
2. The method according to claim 1, characterized in that The image features include respective regional features of a plurality of regional images corresponding to the pedestrian detection frame diagram; The method of using the pre-trained first model to determine the image features corresponding to each of the pedestrian detection block diagrams, and inputting the image features into the pre-trained second model to extract human key points, and determining the key point features of each human key point area in each of the pedestrian detection block diagrams and the confidence corresponding to the key point features, includes: Determine, by using the first model, respective regional features of a plurality of regional images corresponding to each of the pedestrian detection frame images; Inputting the regional features of the multiple regional images of each of the pedestrian detection frame diagrams into the second model to extract human key points, and determining the key point features of each human key point region in the multiple regional images of each of the pedestrian detection frame diagrams and the confidence corresponding to the key point features; The step of fusing the key point features, the confidence level and the image features to obtain fusion features corresponding to each key point region of the human body in each pedestrian detection frame diagram includes: The key point features, the confidence levels and the regional features are subjected to feature fusion to obtain fusion features corresponding to the key point regions of the human body in the plurality of regional images of each pedestrian detection frame diagram.
3. The method according to claim 2, characterized in that The using the first model to determine the respective regional features of the plurality of regional images corresponding to each of the pedestrian detection frame diagrams includes: Using the first model, the pedestrian detection frame diagram is divided into a plurality of region images, and a target sub-region image is intercepted from each of the region images; Feature extraction is performed on the multiple target sub-region images to obtain region features corresponding to the multiple region images.
4. The method according to claim 2 or 3, characterized in that: The step of inputting the regional features of the multiple regional images of each of the pedestrian detection frame diagrams into the second model to extract key points of the human body, and determining the key point features of each key point region of the human body in the multiple regional images of each of the pedestrian detection frame diagrams and the confidence corresponding to the key point features, comprises: Inputting the regional features of the plurality of regional images of each of the pedestrian detection frame diagrams into the second model, performing at least one convolution process on each of the regional features to obtain convolution features, and using the convolution features to extract human key points to obtain key point features of each human key point region corresponding to each of the regional images; Convolution processing is performed on each of the key point features at least once, and the processed features are normalized to obtain the confidence corresponding to each of the key point features.
5. The method according to claim 2, characterized in that: The step of fusing the key point features, the confidence level and the regional features to obtain fusion features corresponding to the key point regions of the human body in the plurality of regional images of each pedestrian detection frame diagram includes: For each of the human key point regions in each of the regional images, weighted processing is performed on the key point features and the confidence of the human key point region, and feature fusion is performed on the weighted result and the key point features to obtain a first feature; Performing feature connection processing on the regional feature and the first feature to obtain a second feature; The convolutional attention module is used to perform feature fusion on the second feature and the regional feature to obtain the fused feature of the human body key point area.
6. The method according to claim 1 or 2, characterized in that: The pedestrian recognition result includes the pedestrian's identification mark; Determining the pedestrian recognition result corresponding to each pedestrian detection frame diagram according to the fusion features of each human key point area in the pedestrian detection frame diagram corresponding to the heterogeneous cameras respectively includes: Determining a reference camera and a matching camera among the heterogeneous cameras; For any first pedestrian detection frame image corresponding to the reference camera, determine the similarity between the first pedestrian detection frame image and the second pedestrian in each of the second pedestrian detection frame images according to the fusion features corresponding to the human body key point regions in the first pedestrian detection frame image and the fusion features of the human body key point regions in each second pedestrian detection frame image corresponding to the matching camera; Determining a matching result between the first pedestrian and each of the second pedestrians according to the similarity; the matching result is used to indicate whether the second pedestrian is the same as the first pedestrian; According to the matching result of the first pedestrian in each first pedestrian detection frame corresponding to the reference camera, the identification mark of each first pedestrian and the identification mark of each second pedestrian are determined.
7. The method according to claim 1, characterized in that The heterogeneous cameras include a wide-angle camera and a fisheye camera; Before acquiring the images including pedestrians respectively collected by the vehicle-mounted heterogeneous cameras, the method further includes: Acquire a first image including pedestrians taken by the wide-angle camera and a second image including pedestrians taken by the fisheye camera; Performing a dedistortion process on the second image to obtain a third image corresponding to the fisheye camera; The performing pedestrian detection on the image and cropping the image according to the pedestrian detection frame to obtain a plurality of pedestrian detection frame diagrams includes: Pedestrian detection is performed on the first image and the third image respectively, and the first image and the third image are cropped respectively according to the pedestrian detection frame to obtain multiple pedestrian detection frame diagrams.
8. The method according to claim 1, characterized in that The first model is trained according to the following steps: Acquire a sample image data set; the sample image data set includes a plurality of sample image pairs, and the sample image pairs include sample images respectively taken by the heterogeneous cameras; For any of the sample image pairs, a sample pedestrian detection block diagram in each sample image included in the sample image pair is input into a first model to be trained to obtain prediction region features corresponding to a plurality of sample region images in the sample pedestrian detection block diagram; Merging the predicted region features and the manually constructed shared features corresponding to the sample region image, and inputting the merged features into a decoding module in the first model to be trained to obtain a restored region image corresponding to the sample region image; Determine a first loss of a first model to be trained according to each of the sample region images corresponding to each of the sample image pairs and the restored region image corresponding to each of the sample region images; The first model to be trained is iteratively trained using the first loss until a training cutoff condition is met, thereby obtaining a trained first model.
9. The method according to claim 1, characterized in that: The method further comprises the step of training the second model: Obtaining sample area features of each sample area image in a plurality of sample pedestrian detection frame diagrams; the sample area features are obtained by extracting features of the sample pedestrian detection frame diagram in the sample image using a trained first model; the sample image is an image in a sample image pair; the sample image pair includes sample images respectively taken by the heterogeneous cameras; Input the sample area features into the second model to be trained to extract human key points, determine the predicted key point features of each human key point area corresponding to each sample area image in each sample pedestrian detection frame diagram and the prediction confidence corresponding to the predicted key point features; Determining the prediction category of each pixel point in each of the sample area images by using the prediction key point features and the prediction confidence; Performing feature fusion on the predicted key point features, the predicted confidence and the sample area features to obtain predicted fusion features corresponding to each human key point area in each sample area image in each sample pedestrian detection frame diagram, and determining the predicted identification of the pedestrian in each sample pedestrian detection frame diagram according to the predicted fusion features; According to the predicted category and label category of each pixel point in the sample pedestrian detection frame diagram, the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram, and the predicted fusion feature, a second loss of the second model to be trained is determined, and the second model to be trained is iteratively trained using the second loss until a training cutoff condition is met to obtain a trained second model.
10. The method according to claim 9, characterized in that The determining, according to the predicted category and label category of each pixel point in the sample pedestrian detection frame diagram, the predicted identifier and label identifier of the pedestrian in the sample pedestrian detection frame diagram, and the predicted fusion feature, a second loss of the second model to be trained comprises: Determining a key point prediction loss according to a prediction category and a label category of each pixel point in the sample pedestrian detection frame map; Determining recognition loss according to the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram and the predicted fusion feature; The second loss is determined according to the key point prediction loss and the recognition loss.
11. The method according to claim 10, characterized in that The determining of the recognition loss according to the predicted identification and label identification of the pedestrian in the sample pedestrian detection frame diagram and the predicted fusion feature includes: Determine a marker prediction loss according to the predicted markers of the pedestrians in each of the sample pedestrian detection frame diagrams, the label marker of each of the pedestrians, and the first number of the pedestrians; For any first sample pedestrian detection frame diagram in each of the sample pedestrian detection frame diagrams, according to a first label identifier of a pedestrian in the first sample pedestrian detection frame diagram and a second label identifier of a pedestrian in each second sample pedestrian detection frame diagram except the first sample pedestrian detection frame diagram, determining a positive sample set and a negative sample set corresponding to the first sample pedestrian detection frame diagram from the second sample pedestrian detection frame diagram; Determine the triplet loss by using the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the positive sample set, and the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the negative sample set; The recognition loss is determined according to the identification prediction loss and the triplet loss.
12. The method according to claim 11, characterized in that The method of determining a positive sample set and a negative sample set corresponding to the first sample pedestrian detection frame diagram from the second sample pedestrian detection frame diagram according to the first label identification of the pedestrian in the first sample pedestrian detection frame diagram and the second label identification of the pedestrian in each second sample pedestrian detection frame diagram except the first sample pedestrian detection frame diagram, comprises: Selecting a second number of target pedestrians from the first number of pedestrians; For each of the target pedestrians, according to the third label identifier of the target pedestrian and each of the second label identifiers, a third number of third sample pedestrian detection frame diagrams corresponding to each of the target pedestrians are screened out from each of the second sample pedestrian detection frame diagrams; According to the first label identification and the third label identification of each target pedestrian, a set of positive samples with the same identification as that corresponding to the first sample pedestrian detection frame diagram and a set of negative samples with different identifications corresponding to the first sample pedestrian detection frame diagram are determined from the third sample pedestrian detection frame diagram corresponding to each target pedestrian.
13. The method according to claim 11, characterized in that The method of determining the triplet loss by using the predicted fusion features of each human key point area corresponding to the first sample pedestrian detection frame diagram, the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the positive sample set, and the predicted fusion features of each human key point area corresponding to each second sample pedestrian detection frame diagram in the negative sample set comprises: For any second sample pedestrian detection frame diagram in the positive sample set, selecting a plurality of first target features from the predicted fusion features of each human key point area corresponding to the second sample pedestrian detection frame diagram; For any second sample pedestrian detection frame diagram in the negative sample set, selecting a plurality of second target features from the predicted fusion features of each human key point area corresponding to the second sample pedestrian detection frame diagram; Selecting, from the predicted fusion features of each human key point region corresponding to the first sample pedestrian detection frame diagram, a third target feature matching each of the first target features and a fourth target feature matching each of the second target features; The triplet loss is determined according to the Euclidean distance between each of the first target features and the matched third target feature, and the Euclidean distance between each of the second target features and the matched fourth target feature.
14. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the pedestrian re-identification method as described in any one of claims 1 to 13.
15. A computer program product, wherein when the computer program product is executed by a computer device, the computer device executes the steps of the pedestrian re-identification method according to any one of claims 1 to 13.
Citation Information
Cited By
Blind zone detection method, vehicle, readable storage medium and computer program product
CN121084375A
