Pedestrian re-identification method, device and equipment based on multimodal semantic information

Abstract

The present disclosure provides a method, device and equipment for pedestrian re-identification based on multimodal semantic information, the method comprising: obtaining a number of monitoring screens; using the number of monitoring screens as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring screen, and determines the appearance feature labels corresponding to each pedestrian and the depth features corresponding to each appearance feature label; determining pedestrians with the same appearance feature labels as the target retrieval object as persons to be identified; determining the movement trajectories of the target retrieval object and each person to be identified in their respective monitoring screens based on the target tracking algorithm, and determining the trajectory similarity between the target retrieval object and each person to be identified; determining the person to be identified whose trajectory similarity is greater than or equal to a predetermined threshold as the target retrieval object. The technical solution of the embodiment of the present application can improve the efficiency of pedestrian re-identification while ensuring the accuracy of pedestrian re-identification.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and in particular relates to a pedestrian re-identification method, device and equipment based on multimodal semantic information. Background Art

[0002] Person re-identification refers to matching images of the same person in different pictures to identify the identity and route of the target in the picture. It is widely used in automated personnel tracking scenarios and plays an increasingly important role in security monitoring, crowd analysis, intelligent transportation and other fields. In the current technical solutions, some pedestrian re-identification methods combine multiple features of pedestrians (such as color, texture, motion and other information) to perform pedestrian recognition. However, the above methods often require complex data preprocessing and feature engineering, and are prone to information loss or inconsistency when fusing different modal information. Therefore, how to improve the efficiency of pedestrian re-identification while ensuring the accuracy of pedestrian re-identification has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus and device for pedestrian re-identification based on multimodal semantic information, which can improve the efficiency of pedestrian re-identification while ensuring the accuracy of pedestrian re-identification, at least to a certain extent.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a pedestrian re-identification method based on multimodal semantic information is provided, comprising:

[0006] Acquire several monitoring images obtained by shooting the target area;

[0007] Using the plurality of monitoring images as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring image, determines the pedestrians contained therein and extracts a set of deep features corresponding to each pedestrian, performs recognition according to the set of deep features corresponding to each pedestrian, determines an appearance feature label corresponding to each pedestrian and a depth feature corresponding to each appearance feature label;

[0008] Determine a pedestrian who has all the same appearance feature labels as a target search object as a person to be identified, wherein the target search object is specified from any pedestrian included in the monitoring screen;

[0009] Based on the target tracking algorithm, determine the movement trajectory of the target retrieval object and each person to be identified in the respective monitoring screen;

[0010] Determine the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory;

[0011] The persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold are determined as target retrieval objects.

[0012] According to one aspect of an embodiment of the present application, a pedestrian re-identification device based on multimodal semantic information is provided, comprising:

[0013] An acquisition module is used to acquire a number of monitoring images obtained by shooting a target area;

[0014] An extraction module, used to use the plurality of monitoring images as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring image, determines the pedestrians contained therein and extracts a depth feature set corresponding to each pedestrian, performs recognition according to the depth feature set corresponding to each pedestrian, determines an appearance feature label corresponding to each pedestrian and a depth feature corresponding to each appearance feature label;

[0015] A screening module, used to determine pedestrians with the same appearance feature labels as the target search object as the person to be identified, wherein the target search object is specified from the pedestrians included in any of the monitoring images;

[0016] A tracking module, used to determine the movement trajectory of the target retrieval object and each person to be identified in the respective monitoring screen based on a target tracking algorithm;

[0017] A calculation module, used to determine the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory;

[0018] The processing module is used to determine the persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold as target retrieval objects.

[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for pedestrian re-identification based on multimodal semantic information as described in the above embodiment is implemented.

[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the pedestrian re-identification method based on multimodal semantic information as described in the above embodiments.

[0021] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the pedestrian re-identification method based on multimodal semantic information provided in the above embodiments.

[0022] In the technical solutions provided in some embodiments of the present application, a number of monitoring screens are obtained by photographing a target area, and the number of monitoring screens are used as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection target model detects according to each monitoring screen, determines the pedestrians contained therein, and extracts the depth feature set corresponding to each pedestrian. Then, recognition is performed according to the depth feature set corresponding to each pedestrian, and the appearance feature label corresponding to each pedestrian and the depth feature corresponding to each appearance feature label are determined; first, a target retrieval object is specified from the pedestrians contained in any monitoring screen, and pedestrians with the same appearance feature labels as the target retrieval object are determined as persons to be identified. Then, based on the target tracking algorithm, the target retrieval object is determined. The motion trajectories of the target object and the persons to be identified in their respective monitoring images are determined according to all the appearance feature labels and depth features of the target retrieval object on its motion trajectory, as well as all the appearance feature labels and depth features of the persons to be identified on their motion trajectory. The trajectory similarity between the target retrieval object and the persons to be identified is determined, and the persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold are determined as the target retrieval object. In this way, based on the multimodal detection model, the appearance feature labels and corresponding depth features of all pedestrians can be extracted after one detection, and this is used for preliminary screening and final determination of pedestrian re-identification. There is no need for complex data preprocessing and feature engineering, which improves the efficiency of pedestrian re-identification while ensuring the accuracy of pedestrian re-identification.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 A schematic diagram of a process of a pedestrian re-identification method based on multimodal semantic information according to an embodiment of the present application is shown;

[0026] Figure 2 A block diagram of a pedestrian re-identification device based on multimodal semantic information according to an embodiment of the present application is shown;

[0027] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0029] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0030] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0032] Figure 1 A schematic flow chart of a pedestrian re-identification method based on multimodal semantic information according to an embodiment of the present application is shown.

[0033] The method can be applied to a terminal device or a server, wherein the terminal device may include but is not limited to one or more of a smart phone, a tablet computer, a portable computer, and a desktop computer; the server may be a physical server or a cloud server.

[0034] like Figure 1 As shown, the pedestrian re-identification method based on multimodal semantic information includes at least steps S110 to S160, which are described in detail as follows (hereinafter, the method is described by taking the application of the method to a terminal device as an example, hereinafter referred to as "terminal"):

[0035] In step S110, a plurality of monitoring images obtained by photographing the target area are acquired.

[0036] The target area may be an area with video surveillance, such as a public place such as a railway station, a square, a shopping mall, etc. The target area may be provided with one or more cameras to capture image data of roads and pedestrians in the target area.

[0037] In this embodiment, the terminal can communicate with one or more cameras arranged corresponding to the target area, thereby obtaining several monitoring images of each camera.

[0038] In one embodiment, when pedestrian re-identification is not required, the surveillance images captured by the camera can be stored first and overwritten according to a predetermined time period. When pedestrian re-identification is required, the user can specify the time and space range of pedestrian re-identification through the terminal, that is, the user can specify a time period for retrieval and a specific target area for retrieval, for example, the retrieval range can be set to all cameras in the mall this morning, etc. In this way, the retrieval range can be clarified, useless retrieval can be avoided, and the efficiency of pedestrian re-identification can be improved.

[0039] In step S120, several of the monitoring images are used as input to a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection based on each of the monitoring images, determines the pedestrians contained therein and extracts the depth feature set corresponding to each of the pedestrians, and identifies each pedestrian based on the depth feature set corresponding to each of the pedestrians, determines the appearance feature labels corresponding to each of the pedestrians and the depth features corresponding to each of the appearance feature labels.

[0040] In this embodiment, a person skilled in the art may pre-build and train a multimodal target detection model, such as a Yolo-World model or an owlv2 model, etc. When pedestrian re-identification is required, the terminal may use the acquired monitoring images as inputs of the multimodal target detection model, and the multimodal target detection model may perform detection based on each monitoring image, thereby identifying all pedestrians in the target area within the corresponding search range, and extracting a depth feature set corresponding to each pedestrian, wherein the depth feature set includes at least one depth feature.

[0041] Next, the multimodal target detection model can identify the appearance characteristics of each pedestrian, such as clothing and body shape, based on the corresponding deep feature set of each pedestrian, thereby generating corresponding appearance feature labels and storing the deep features corresponding to each appearance feature label.

[0042] In step S130, pedestrians having the same appearance feature labels as the target search object are determined as persons to be identified, and the target search object is specified from any pedestrian included in the monitoring screen.

[0043] In this embodiment, when performing pedestrian re-identification, the user can specify one person from the pedestrians included in the plurality of monitoring images as the target search object. After step S120, the terminal can match the appearance feature label corresponding to the target search object, and determine the pedestrian whose appearance feature label is exactly the same as the target search object as the person to be identified, so as to complete the initial screening. In this way, the person to be identified is the pedestrian target with a similar appearance to the target search object under different cameras.

[0044] In step S140, based on the target tracking algorithm, the movement trajectories of the target retrieval object and each of the persons to be identified in their respective monitoring images are determined.

[0045] In this embodiment, after screening out the persons to be identified, the terminal can use a target tracking algorithm (such as a deepsort tracking algorithm, etc.) to calculate the target retrieval object and the motion trajectory of each person to be identified in their respective monitoring images, and group all the appearance feature labels and depth features of the same person on the motion trajectory as a group to participate in the final pedestrian re-identification.

[0046] In step S150, the trajectory similarity between the target retrieval object and each of the persons to be identified is determined based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, as well as all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory.

[0047] In this embodiment, the terminal can determine the trajectory similarity between the target retrieval object and each person to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each person to be identified on its motion trajectory. It should be noted that the greater the trajectory similarity, the higher the possibility that the two are the same person, and vice versa, the smaller the trajectory similarity, the lower the possibility that the two are the same person.

[0048] In one embodiment, according to all appearance feature labels and depth features of the target search object on its motion trajectory, and all appearance feature labels and depth features of each person to be identified on their motion trajectory, determining the trajectory similarity between the target search object and each person to be identified includes:

[0049] According to the appearance feature label and depth feature of the target search object on its motion trajectory, and the appearance feature label and depth feature of each person to be identified on their motion trajectory, the cosine distance between the depth features corresponding to the same appearance feature label of the two is calculated;

[0050] The trajectory similarity between the target retrieval object and each of the persons to be identified is determined according to the cosine distance corresponding to each of the appearance feature labels of each of the persons to be identified.

[0051] In this embodiment, the terminal can use the appearance feature labels and depth features of the target retrieval object on its motion trajectory as a benchmark, and perform cosine distance calculation on the depth features corresponding to the same appearance feature labels in other screened motion trajectories. In this way, for each person to be identified, one appearance feature label corresponds to one cosine distance, and the terminal can comprehensively consider the cosine distances of all appearance feature labels of the person to be identified to obtain the trajectory similarity between the target retrieval object and the person to be identified.

[0052] In one example, the terminal may average the cosine distances of all appearance feature labels of the person to be identified, thereby obtaining the final trajectory similarity of the person to be identified. It should be noted that other methods, such as summation or weighted sum calculation methods, may also be used to obtain the trajectory similarity corresponding to the person to be identified, and this application does not make any special restrictions on this.

[0053] In one example, if there are multiple depth features corresponding to the same appearance feature label in a certain motion trajectory, the multiple depth features can be averaged before performing subsequent cosine distance calculation.

[0054] Please continue to refer to Figure 1 In step S160, the persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold are determined as target retrieval objects.

[0055] In this embodiment, after obtaining the trajectory similarity corresponding to each person to be identified, the terminal can compare each trajectory similarity with a predetermined threshold, which can be pre-set by a person skilled in the art based on prior experience. When the trajectory similarity is greater than or equal to the predetermined threshold, it can be determined that the person to be identified and the target search object are the same person, and the motion trajectory of the person to be identified is the motion trajectory of the target search object.

[0056] In this way, the terminal can perform trajectory analysis based on the movement trajectory of the person to be identified who is determined as the target retrieval object, so as to determine the last location where the target retrieval object appeared.

[0057] In some embodiments of the present application, when training a multimodal target detection model, the following steps are included:

[0058] Acquire several training images;

[0059] Using the plurality of training images and the plurality of prompt words as inputs of a multimodal object detection model to be trained, so that the multimodal object detection model recognizes an image region corresponding to each of the prompt words from the training images, and determines the corresponding image region feature embedding;

[0060] Packing and storing each of the prompt words and its corresponding image region feature embedding as a training feature set;

[0061] The training feature sets corresponding to the same person in different training pictures are taken as a group, and the data corresponding to the same prompt words in different groups are trained using the triplet learning method, so that the distance between the features within the group is shortened while the distance between the features between groups is increased.

[0062] In this embodiment, after the multimodal target detection model is constructed, the multimodal target detection model can be trained to improve its performance and ensure its recognition accuracy. Specifically, the terminal can obtain a number of training images, which can be obtained by shooting the monitoring area with different cameras, so as to capture the full body image of the pedestrian from different angles, ensuring the diversity and comprehensiveness of the data.

[0063] In one example, the terminal may first perform basic equalization processing such as noise reduction, white balance, and brightness adjustment on the acquired training picture before performing subsequent training steps.

[0064] Next, the terminal can use the equalized training images and the prompt words as inputs of the multimodal target detection model to be trained, so that the multimodal target detection model can identify the image area corresponding to each prompt word from the training images and determine the corresponding image area feature embedding.

[0065] Taking the Yolo-World model as an example of a multimodal target detection model, first, we can set as many prompt words as possible that need to be detected, such as "pedestrian, male, white shirt, backpack", etc., and use the text vectorization model based on the BERT algorithm to calculate them separately, and output the results as text features; secondly, send the training picture to the image feature extraction backbone network intercepted from the yolov8 algorithm, perform neural network operations at three scales, and output three feature maps; finally, through the image-text feature matching network, calculate the area that meets the text features on the feature map of the image, and use it as the detected target, while saving the detection category, detection box, confidence, and corresponding local feature matrix information, and generate the corresponding image area feature embedding.

[0066] In this way, by setting multiple levels of prompt words at the algorithm model level, only one reasoning can capture the location (detection frame) of all pedestrians in the picture, as well as visual features such as clothing and body shape and the corresponding image range. By adjusting the prompt words and confidence filtering, a structured description of pedestrians can be formed from multiple aspects. And by flexibly setting prompt words, the model of this application can adapt to different monitoring scenarios, such as clothing customs in different regions, etc., which improves the adaptability of the method.

[0067] Next, the image region feature embeddings corresponding to each prompt word machine are packaged and stored as a training feature set to serve as a basis for subsequent matching.

[0068] In order to improve the accuracy of model testing and the discrimination between different targets, we can manually organize the data and group the training feature sets of the same person in different training pictures as a group. For the data of the same prompt words in different groups, the model training method is used for triple learning, so that the distance between features within the group is shortened while the distance between features between groups is increased, which facilitates the obtaining of greater discrimination between groups in the subsequent feature matching stage, thus being more conducive to re-identification.

[0069] In one embodiment, the data corresponding to the same prompt word in different groups are trained by using a triplet learning method to make the feature distance within the group closer while the feature distance between the groups farther apart, including:

[0070] During the model training process, three samples are selected each time, including an anchor sample, a positive sample, and a negative sample, wherein the anchor sample and the positive sample correspond to the same person, and the negative sample corresponds to another person;

[0071] After extracting features using the current model, respectively calculating the cosine distance between the anchor point sample and the positive sample, and the cosine distance between the anchor point sample and the negative sample in the feature space;

[0072] Constructing a target loss function, wherein the target loss function includes a triple loss function and an InfoNCE loss function of contrastive learning;

[0073] The multimodal target detection model is trained according to the target loss function until convergence.

[0074] In this embodiment, triplet learning is a deep learning training strategy. During the model training process, three samples are selected each time: an anchor sample, a positive sample, and a negative sample. The anchor sample and the positive sample belong to the same pedestrian, while the negative sample belongs to another pedestrian. After extracting features using the current model, the cosine similarity is used to calculate the distance between the anchor sample and the positive sample in the feature space, as well as the distance between the anchor sample and the negative sample. Since the inter-group distance and inter-class distance of the multimodal model need to be considered at the same time, it is necessary to combine triplet learning with the original contrastive learning of the model, so a triplet loss function and contrastive learning are proposed. InfoNCE The objective loss function of the loss function is used as the loss function for model training.

[0075] Because the target loss function combines the triplet loss and the multimodal contrast loss, it can be called a multimodal triplet loss function. The target loss function is obtained in each round of training. L total After obtaining the results, the gradient descent and back propagation algorithms can be used to adjust the model parameters, thereby reducing the target loss function in the next round of training and finally converging it to an acceptable range of values. This process will make the feature vectors of the same pedestrian closer in the feature space, and the feature vectors of different pedestrians farther away, while ensuring that the target detection algorithm can correctly detect targets by category, expanding the capabilities of the multimodal model.

[0076] In one embodiment, the objective loss function is constructed according to the following formula:

[0077] ,in, L triplet is the triplet loss function, L InfoNCE It is comparative learning InfoNCE Loss function, λ is the weight coefficient.

[0078] In one example, the triple loss function is constructed according to the following formula:

[0079]

[0080] in, It isi Anchor point samples a i With i Positive samples p i The cosine distance between It is i Anchor point samples a i With i Negative samples n i The cosine distance between margin is a hyperparameter used to define the minimum distance difference between positive and negative sample pairs. N is the number of samples in the triplet.

[0081] In one example, the following formula is used to construct InfoNCE Loss function:

[0082]

[0083] in, I i It is i The feature representation of the training pictures is T i is with I i The feature representation of positively correlated prompt words, T j It is the first in the batch j The feature representation of the prompt word, sim ( , ) is a function that calculates the similarity between two feature representations, τ is the temperature parameter used to control InfoNCE In loss softmax distributed, M is the number of training frames in a batch.

[0084] It should be noted that based on the above training process, through the zero-sample detection capability of the multimodal target detection model, the present invention gets rid of the need for a large amount of labeled data in traditional re-identification technology, reduces the cost and workload of data preparation, and also protects the privacy of pedestrians.

[0085] The following describes an apparatus embodiment of the present application, which can be used to execute the pedestrian re-identification method based on multimodal semantic information in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the pedestrian re-identification method based on multimodal semantic information in the present application.

[0086] Figure 2 A block diagram of a pedestrian re-identification device based on multimodal semantic information according to an embodiment of the present application is shown.

[0087] Reference Figure 2 As shown, according to an embodiment of the present application, a pedestrian re-identification device based on multimodal semantic information includes:

[0088] An acquisition module is used to acquire a number of monitoring images obtained by shooting a target area;

[0089] An extraction module, used to use the plurality of monitoring images as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring image, determines the pedestrians contained therein and extracts a depth feature set corresponding to each pedestrian, performs recognition according to the depth feature set corresponding to each pedestrian, determines an appearance feature label corresponding to each pedestrian and a depth feature corresponding to each appearance feature label;

[0090] A screening module, used to determine pedestrians with the same appearance feature labels as the target search object as the person to be identified, wherein the target search object is specified from the pedestrians included in any of the monitoring images;

[0091] A tracking module, used to determine the movement trajectory of the target retrieval object and each person to be identified in the respective monitoring screen based on a target tracking algorithm;

[0092] A calculation module, used to determine the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory;

[0093] The processing module is used to determine the persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold as target retrieval objects.

[0094] In one embodiment, according to all appearance feature labels and depth features of the target search object on its motion trajectory, and all appearance feature labels and depth features of each person to be identified on their motion trajectory, determining the trajectory similarity between the target search object and each person to be identified includes:

[0095] According to the appearance feature label and depth feature of the target search object on its motion trajectory, and the appearance feature label and depth feature of each person to be identified on their motion trajectory, the cosine distance between the depth features corresponding to the same appearance feature label of the two is calculated;

[0096] The trajectory similarity between the target retrieval object and each of the persons to be identified is determined according to the cosine distance corresponding to each of the appearance feature labels of each of the persons to be identified.

[0097] In one embodiment, when training the multimodal target detection model, the processing module is further used to:

[0098] Acquire several training images;

[0099] Using the plurality of training images and the plurality of prompt words as inputs of a multimodal object detection model to be trained, so that the multimodal object detection model recognizes an image region corresponding to each of the prompt words from the training images, and determines the corresponding image region feature embedding;

[0100] Packing and storing each of the prompt words and its corresponding image region feature embedding as a training feature set;

[0101] The training feature sets corresponding to the same person in different training pictures are taken as a group, and the data corresponding to the same prompt words in different groups are trained using the triplet learning method, so that the distance between the features within the group is shortened while the distance between the features between groups is increased.

[0102] In one embodiment, the data corresponding to the same prompt word in different groups are trained by using a triplet learning method to make the feature distance within the group closer while the feature distance between the groups farther apart, including:

[0103] During the model training process, three samples are selected each time, including an anchor sample, a positive sample, and a negative sample, wherein the anchor sample and the positive sample correspond to the same person, and the negative sample corresponds to another person;

[0104] After extracting features using the current model, respectively calculating the cosine distance between the anchor point sample and the positive sample, and the cosine distance between the anchor point sample and the negative sample in the feature space;

[0105] Construct a target loss function, which includes a triple loss function and a contrastive learning loss function. InfoNCE Loss function;

[0106] The multimodal target detection model is trained according to the target loss function until convergence.

[0107] Figure 3 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0108] It should be noted that Figure 3 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 3As shown, the computer system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0110] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0111] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, various functions defined in the system of the present application are executed.

[0112] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0113] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0115] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0116] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0117] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0118] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0119] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A pedestrian re-identification method based on multimodal semantic information, characterized in that: include: Acquire several monitoring images obtained by shooting the target area; Using the plurality of monitoring images as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring image, determines the pedestrians contained therein and extracts a set of deep features corresponding to each pedestrian, performs recognition according to the set of deep features corresponding to each pedestrian, determines an appearance feature label corresponding to each pedestrian and a depth feature corresponding to each appearance feature label; Determine a pedestrian who has all the same appearance feature labels as a target search object as a person to be identified, wherein the target search object is specified from any pedestrian included in the monitoring screen; Based on the target tracking algorithm, determine the movement trajectory of the target retrieval object and each person to be identified in the respective monitoring screen; Determine the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory; Determine the person to be identified whose trajectory similarity is greater than or equal to a predetermined threshold as a target retrieval object; When training the multimodal target detection model, it includes: Acquire several training images; Using the plurality of training images and the plurality of prompt words as inputs of a multimodal object detection model to be trained, so that the multimodal object detection model recognizes an image region corresponding to each of the prompt words from the training images, and determines the corresponding image region feature embedding; Packing and storing each of the prompt words and its corresponding image region feature embedding as a training feature set; The training feature sets corresponding to the same person in different training pictures are taken as a group, and the data corresponding to the same prompt words in different groups are trained using the triplet learning method, so that the distance between the features within the group is shortened while the distance between the features between groups is increased.

2. The method according to claim 1, characterized in that Determining the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory, includes: According to the appearance feature label and depth feature of the target search object on its motion trajectory, and the appearance feature label and depth feature of each person to be identified on their motion trajectory, the cosine distance between the depth features corresponding to the same appearance feature label of the two is calculated; The trajectory similarity between the target retrieval object and each of the persons to be identified is determined according to the cosine distance corresponding to each of the appearance feature labels of each of the persons to be identified.

3. The method according to claim 1, characterized in that The data corresponding to the same prompt word in different groups are trained using a triplet learning method to make the distance between the features within the group closer while the distance between the features between groups farther, including: During the model training process, three samples are selected each time, including an anchor sample, a positive sample, and a negative sample, wherein the anchor sample and the positive sample correspond to the same person, and the negative sample corresponds to another person; After extracting features using the current model, respectively calculating the cosine distance between the anchor point sample and the positive sample, and the cosine distance between the anchor point sample and the negative sample in the feature space; Constructing a target loss function, wherein the target loss function includes a triple loss function and an InfoNCE loss function of contrastive learning; The multimodal target detection model is trained according to the target loss function until convergence.

4. The method according to claim 3, characterized in that The objective loss function is constructed according to the following formula: THE total =λL triplet +(1-λ)L InfoNCE Among them, L triplet is the triplet loss function, L InfoNCE is the InfoNCE loss function of contrastive learning, and λ is the weight coefficient.

5. The method according to claim 4, characterized in that The triplet loss function is constructed according to the following formula: Among them, d(a i ,p i ) is the i-th anchor point sample a i With the i-th positive sample p i The cosine distance between i ,n i ) is the i-th anchor point sample a i With the i-th negative sample n i The cosine distance between them, margin is a hyperparameter, and N is the number of samples in the triplet.

6. The method according to claim 4, characterized in that The InfoNCE loss function is constructed according to the following formula: Among them, I i is the feature representation of the i-th training picture, T i Yes and I i The feature representation of positively correlated prompt words, T j is the feature representation of the jth cue word in the batch, sim(,) is the function that calculates the similarity between two feature representations, τ is the temperature parameter used to control the softmax distribution in the InfoNCE loss, and M is the number of training images in the batch.

7. A pedestrian re-identification device based on multimodal semantic information, characterized in that: include: An acquisition module is used to acquire a number of monitoring images obtained by shooting a target area; An extraction module, used to use the plurality of monitoring images as inputs of a pre-trained multimodal target detection model, so that the multimodal target detection model performs detection according to each monitoring image, determines the pedestrians contained therein and extracts a depth feature set corresponding to each pedestrian, performs recognition according to the depth feature set corresponding to each pedestrian, determines an appearance feature label corresponding to each pedestrian and a depth feature corresponding to each appearance feature label; A screening module, used to determine pedestrians with the same appearance feature labels as the target search object as the person to be identified, wherein the target search object is specified from the pedestrians included in any of the monitoring images; A tracking module, used to determine the movement trajectory of the target retrieval object and each person to be identified in the respective monitoring screen based on a target tracking algorithm; A calculation module, used to determine the trajectory similarity between the target retrieval object and each of the persons to be identified based on all appearance feature labels and depth features of the target retrieval object on its motion trajectory, and all appearance feature labels and depth features of each of the persons to be identified on their motion trajectory; A processing module, used for determining the persons to be identified whose trajectory similarity is greater than or equal to a predetermined threshold as target retrieval objects; When training the multimodal target detection model, the processing module is also used to: Acquire several training images; Using the plurality of training images and the plurality of prompt words as inputs of a multimodal object detection model to be trained, so that the multimodal object detection model recognizes an image region corresponding to each of the prompt words from the training images, and determines the corresponding image region feature embedding; Packing and storing each of the prompt words and its corresponding image region feature embedding as a training feature set; The training feature sets corresponding to the same person in different training pictures are taken as a group, and the data corresponding to the same prompt words in different groups are trained using the triplet learning method, so that the distance between the features within the group is shortened while the distance between the features between groups is increased.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for pedestrian re-identification based on multimodal semantic information as claimed in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the pedestrian re-identification method based on multimodal semantic information as described in any one of claims 1 to 6.

Images