Multi-mode ReID detection method and device based on transfer learning, medium and equipment

Abstract

The invention discloses a multi-modal ReID detection method and device based on transfer learning, a medium and equipment, and the method comprises the steps: obtaining an original data set based on the same scene, carrying out the model training based on the original data set, and obtaining a multi-modal module; the method comprises the following steps: acquiring to-be-detected video stream data in real time, analyzing the video stream data, and acquiring single-frame to-be-detected image data; performing face detection processing, face recognition processing, target detection processing, attribute recognition processing and ReID detection processing on the to-be-detected image data; and outputting a pedestrian re-identification detection result. According to the method, the data set is reconstructed through the transfer learning method, the original data set including the light-seeing image data and the infrared image data is obtained, the multi-mode module is obtained through training according to the original data set, and when target detection, attribute recognition and pedestrian re-recognition are carried out, the pedestrian re-recognition efficiency is improved even under the conditions of night vision infrared light, night light and the like. And relatively high detection precision can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-detection, and in particular to a multimodal ReID detection method, device, medium and equipment based on transfer learning. Background Art

[0002] ReID, or person re-identification, is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is widely considered a sub-problem of image retrieval. Given a monitored pedestrian image, retrieve the pedestrian image across devices. It aims to make up for the visual limitations of fixed cameras and can be combined with pedestrian detection / pedestrian tracking technology. It can be widely used in intelligent video surveillance, intelligent security and other fields.

[0003] Due to the differences between different camera devices, and the fact that pedestrians have both rigid and flexible characteristics, and their appearance is easily affected by clothing, scale, occlusion, posture and viewing angle, pedestrian re-identification has become a hot topic in the field of computer vision that is both valuable and challenging.

[0004] In the existing technology, most of the current pedestrian re-identification methods are to perform detection under visible light cameras. However, there is currently no effective detection method for pedestrian re-identification under infrared cameras at night. Compared with single-modal pedestrian re-identification, it is more difficult to achieve pedestrian re-identification under two different modes of visible light and infrared light in all weather conditions day and night. Therefore, how to achieve multi-modal ReID detection is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a multimodal ReID detection method, device, medium and equipment based on transfer learning that overcome the above problems or at least partially solve the above problems.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] According to a first aspect of an embodiment of the present invention, a multimodal ReID detection method based on transfer learning is provided, and the multimodal ReID detection method based on transfer learning includes:

[0008] S1. Acquire an original data set based on the same scene, perform model training based on the original data set, map multiple labeled data in the original data set to the same feature space through a transfer learning method, and then train to obtain a multimodal module;

[0009] S2, acquiring the video stream data to be detected in real time, parsing the video stream data, and acquiring the image data to be detected in a single frame;

[0010] S3, performing face detection processing on the image data to be detected to obtain a first recognition result, confirming whether the first recognition result meets the face quality requirement, and executing step S4 when the first recognition result meets the face quality requirement, and executing step S5 when the first recognition result does not meet the face quality requirement;

[0011] S4, performing face recognition processing on the image data to be detected to obtain a second recognition result, confirming whether the second recognition result matches the face data in the face feature vector database, and when the second recognition result matches the face data in the face feature vector database, executing step S8, and when the second recognition result does not match the face data in the face feature vector database, executing step S5;

[0012] S5, performing target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirming whether the third recognition result meets the target quality requirement, and executing step S6 when the third recognition result meets the target quality requirement; and discarding the image data to be detected and re-executing step S2 when the third recognition result does not meet the target quality requirement;

[0013] S6, performing attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirming whether the fourth recognition result matches the attribute data in the attribute feature vector database, and when the fourth recognition result matches the attribute data in the attribute feature vector database, executing step S8, and when the fourth recognition result does not match the attribute data in the attribute feature vector database, executing step S7;

[0014] S7, performing ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirming whether the fifth recognition result matches the ReID data in the ReID feature vector database, and when the fifth recognition result matches the ReID data in the ReID feature vector database, executing step S8;

[0015] S8. Output a pedestrian re-identification detection result, where the pedestrian re-identification detection result indicates that a pedestrian is successfully identified in the image data to be detected.

[0016] In some embodiments of the present invention, in step S3, performing face detection and recognition processing on the image data to be detected to obtain a first recognition result includes:

[0017] Processing the image data to be detected based on a face detection algorithm, and obtaining face quality parameters after detecting a face;

[0018] Confirm whether the face quality parameters in the image data to be detected meet the face quality requirements, and obtain the first recognition result.

[0019] In some embodiments of the present invention, in step S4, performing face recognition processing on the image data to be detected to obtain a second recognition result includes:

[0020] Processing the image data to be detected based on a face recognition algorithm, extracting face feature data, and obtaining a second recognition result;

[0021] Confirming whether the facial feature data matches the facial data in a facial feature vector database;

[0022] When the facial feature data matches the facial data in the facial feature vector database, executing step S8;

[0023] When the facial feature data does not match the facial data in the facial feature vector database, step S5 is executed, and the facial feature data is stored in the facial feature vector database.

[0024] In some embodiments of the present invention, in step S5, performing target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result includes:

[0025] Performing target detection processing on the image data to be detected, and identifying a target person area from the image data to be detected;

[0026] The image data of the target person area is intercepted, the target quality of the image data of the target person area is confirmed, and a third recognition result is obtained.

[0027] In some embodiments of the present invention, in step S6, performing attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result includes:

[0028] Performing attribute recognition processing on the image data to be detected based on the multimodal module, extracting attribute feature data, and obtaining a fourth recognition result;

[0029] confirming whether the attribute feature data matches the attribute data in the attribute feature vector database;

[0030] When the attribute feature data matches the attribute data in the attribute feature vector database, executing step S8;

[0031] When the attribute feature data does not match the attribute data in the attribute feature vector database, step S7 is executed, and the attribute feature data is stored in the attribute feature vector database.

[0032] In some embodiments of the present invention, in step S7, performing ReID detection processing on the image data to be detected based on the multimodal module to obtain the fifth recognition result includes:

[0033] Performing ReID detection processing on the image data to be detected based on the multimodal module, extracting ReID feature data, and obtaining a fifth recognition result;

[0034] Confirming whether the ReID feature data matches the ReID data in the ReID feature vector database;

[0035] When the ReID feature data matches the ReID data in the ReID feature vector database, executing step S8;

[0036] When the ReID feature data does not match the ReID data in the ReID feature vector database, the ReID feature data is stored in the ReID feature vector database.

[0037] In some embodiments of the present invention, the original data set includes respectively acquired visible light image data of a first target sample, first infrared image data of the same source and second image data of a different source, and acquired third infrared image data of a second target sample.

[0038] According to a second aspect of an embodiment of the present invention, a multimodal ReID detection device based on transfer learning is provided, and the multimodal ReID detection device based on transfer learning includes:

[0039] A training module is used to obtain an original data set based on the same scenario, perform model training based on the original data set, and map multiple labeled data in the original data set to the same feature space through a transfer learning method to obtain a multimodal module through training;

[0040] An image analysis module is used to obtain the video stream data to be detected in real time, analyze the video stream data, and obtain the image data to be detected of a single frame;

[0041] A face detection module is used to perform face detection processing on the image data to be detected, obtain a first recognition result, confirm whether the first recognition result meets the face quality requirement, and output the image data to be detected to the face recognition module when the first recognition result meets the face quality requirement; when the first recognition result does not meet the face quality requirement, output the image data to be detected to the target detection module;

[0042] A face recognition module is used to perform face recognition processing on the image data to be detected to obtain a second recognition result, confirm whether the second recognition result matches the face data in the face feature vector database, and output the image data to be detected to a result output module when the second recognition result matches the face data in the face feature vector database; when the second recognition result does not match the face data in the face feature vector database, output the image data to be detected to a target detection module;

[0043] The target detection module is used to perform target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirm whether the third recognition result meets the target quality requirement, and output the image data to be detected to the attribute recognition module when the third recognition result meets the target quality requirement; when the third recognition result does not meet the target quality requirement, discard the image data to be detected and re-acquire the image data to be detected of the next frame;

[0044] an attribute recognition module, configured to perform attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirm whether the fourth recognition result matches the attribute data in the attribute feature vector database, and output the image data to be detected to a result output module when the fourth recognition result matches the attribute data in the attribute feature vector database, and output the image data to be detected to a ReID detection module when the fourth recognition result does not match the attribute data in the attribute feature vector database;

[0045] A ReID detection module is used to perform ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirm whether the fifth recognition result matches the ReID data in the ReID feature vector database, and execute step S8 when the fifth recognition result matches the ReID data in the ReID feature vector database;

[0046] The result output module is used to output the pedestrian re-identification detection results.

[0047] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which computer program instructions are stored. The computer program instructions are loaded and executed by a processor to implement the operations performed by any of the methods described above.

[0048] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements instructions of any of the methods described above.

[0049] The technical solution provided in the embodiments of the present invention has at least the following technical effects or advantages:

[0050] The embodiments of the present invention provide a multimodal ReID detection method, device, medium and equipment based on transfer learning. The multimodal ReID detection method based on transfer learning described in the embodiments of the present invention reconstructs the data set through the transfer learning method to obtain the original data set including visible light image data and infrared image data. The multimodal module is trained according to the original data set. When performing target detection, attribute recognition and pedestrian re-identification, even in the case of insufficient light such as night vision infrared and night, higher detection accuracy can be obtained, and it has a high prospect for application promotion.

[0051] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a multimodal ReID detection method based on transfer learning provided by an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of the training process principle of the multimodal module;

[0055] Figure 3 A schematic diagram of the principle structure of a multimodal ReID detection device based on transfer learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0057] Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may further design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0058] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element may be directly on the other layer / element, or there may be an intervening layer / element therebetween. Additionally, if a layer / element is "on" another layer / element in one orientation, the layer / element may be "below" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components may be denoted by the same or similar reference numerals.

[0059] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with specific implementation methods. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0060] Figure 1 FIG. 1 is a flow chart of a multimodal ReID detection method based on transfer learning provided by an embodiment of the present invention. Figure 1 As shown, the multimodal ReID detection method based on transfer learning includes the following steps:

[0061] S1. Acquire an original data set based on the same scene, perform model training based on the original data set, map multiple labeled data in the original data set to the same feature space through a transfer learning method, and then train to obtain a multimodal module;

[0062] In an embodiment of the present invention, the original data set includes respectively acquired visible light image data of a first target sample, first infrared image data of the same source and second image data of a different source, and acquired third infrared image data of a second target sample.

[0063] Specifically, an embodiment of the present invention reconstructs a data set based on a transfer learning method to obtain the original data set, including visible light image data and infrared image data; for a first target sample, each time a visible light image is obtained, an infrared image is also obtained, i.e., the visible light image data of the first target sample and the first infrared image data of the same source; and infrared images of the same target sample obtained in different infrared cameras, i.e., the second image data of the different sources; and infrared images of other target samples are obtained at the same time, i.e., the third infrared image data of the second target sample; an embodiment of the present invention maps visible light data and infrared data to the same feature space through a transfer learning method to reduce the difference in feature distribution of visible light data and infrared data.

[0064] See also Figure 2 As shown, it is a schematic diagram of the principle of the training process of the multimodal module. For example, the visible light image data B represents the original annotated visible light data; the visible light image data A represents a small amount of newly annotated visible light data; the first infrared image data a represents the infrared data taken at the same time as the visible light image data A; the second image data a' represents the infrared data of the same sample but different source as the first infrared image data a; the third infrared image data c represents the infrared data of a different sample from the first infrared image data a.

[0065] The embodiment of the present invention uses a feature extractor to extract features of visible light image data and infrared image data, confuses visible light and infrared samples in feature space, and aligns samples of the two modalities for migration.

[0066] The embodiment of the present invention uses a sample classifier to classify a large amount of labeled visible light data and a small amount of labeled infrared data, and to identify the correct labels as much as possible.

[0067] The embodiment of the present invention uses a domain classifier to distinguish whether the data is visible light or infrared.

[0068] The embodiment of the present invention calculates the loss function triplet loss after the feature extractor; the loss function mainly consists of two parts: the training loss of the network (label predictor loss), the triplet loss and the discrimination loss of different domains.

[0069] For the label predictor loss, the calculation process is as follows:

[0070] For the feature extractor G f (Take a single hidden layer as an example), sigmoid is used as the activation function, the input x is the infrared image and the visible light image, W represents the weight in the neural network, b represents the bias term in the neural network, and its output is:

[0071] G f(x; W, b) = sigm(Wx+b);

[0072] For the label predictor G y , softmax is used as the activation function, V represents the first weight in the neural network, c represents the first bias term in the neural network, and its output is:

[0073] G y (G f (x); V,c) = softmax(VG f (x)+c);

[0074] When a data point (x i ,y i ), with negative log-probabality as the loss function, the loss of the label predictor is:

[0075]

[0076] For the triplet loss L t , the calculation process is shown in the following reference:

[0077] triplet loss1=max(d(a,A)-d(a,B)+margin,0)+max(d(a,A)-d(a,c)+margin,0);

[0078] triplet loss2=max(d(a′,a)-d(a′,B)+margin,0)+max(d(a′,a)-d(a′,c)+margin,0);

[0079] L t =triplet loss1+triplet loss2;

[0080] Among them, the function of triplet loss1 is to shorten the distance between the visible light image data and the infrared light image data of the same sample, and the function of triplet loss2 is to shorten the distance between the infrared light image data of the same sample.

[0081] For the discriminative loss of different domains, the calculation process is shown in the following reference:

[0082] For the domain discriminator G d , sigmoid is used as the activation function, u represents the second weight in the neural network, z represents the second bias term in the neural network, and its output is:

[0083] G d (G f(x);u,z)=sigm(u T G f (x)+z);

[0084] The loss of the domain discriminator is as follows:

[0085]

[0086] Where, d i Indicates which modality the data belongs to.

[0087] For the objective function, the calculation process is shown in the following reference:

[0088]

[0089] Wherein, λ is a regularization parameter; in the iterative process, the embodiment of the present invention updates the parameters of the label predictor by minimizing the objective function, and updates the parameters of the domain discriminator by maximizing the objective function.

[0090] In the embodiment of the present invention, a gradient reversal layer GRL is arranged between the feature extractor and the domain discriminator to implement gradient inversion in back propagation so that the feature extractor and the domain discriminator achieve opposite purposes. After adding GRL, the purpose of the feature extractor is to make the domain discriminator unable to correctly determine which mode the data belongs to.

[0091] After the embodiment of the present invention maps multiple labeled data in the original data set to the same feature space through a transfer learning method, the multimodal module is trained to obtain the multimodal module. Since the original data set contains many different types of labeled data, such as visible light image data of a first target sample, first infrared image data of the same source, second image data of a different source, and third infrared image data of an acquired second target sample, high accuracy can be obtained by performing target detection processing, attribute recognition processing and ReID detection processing through the multimodal module in night vision infrared scenarios.

[0092] S2, acquiring the video stream data to be detected in real time, parsing the video stream data, and acquiring the image data to be detected in a single frame;

[0093] The embodiment of the present invention parses the video stream data and performs image capture processing on the video stream data to be detected according to actual needs to obtain a plurality of continuous single-frame image data to be detected. When the image data to be detected in any frame does not meet the requirements, the image data to be detected in the adjacent single-frame can be selected as the new image data to be detected for subsequent recognition processing.

[0094] S3, performing face detection processing on the image data to be detected to obtain a first recognition result, confirming whether the first recognition result meets the face quality requirement, and executing step S4 when the first recognition result meets the face quality requirement, and executing step S5 when the first recognition result does not meet the face quality requirement;

[0095] In an embodiment of the present invention, in step S3, the face detection and recognition processing is performed on the image data to be detected to obtain the first recognition result, including: processing the image data to be detected based on a face detection algorithm, and obtaining face quality parameters after detecting the face; confirming whether the face quality parameters in the image data to be detected meet the face quality requirements, and obtaining the first recognition result.

[0096] The embodiment of the present invention obtains face bbox and face key points through a face detection algorithm (such as retinaface, yolo-face, MTCNN) to make a face quality judgment and confirm whether the first recognition result meets the face quality requirements, wherein the face quality needs to be evaluated by the trained face detection algorithm model for multiple factors such as face size, clarity, pitch angle, deflection angle, etc. When the first recognition result meets the face quality requirements, step S4 is executed, and when the first recognition result does not meet the face quality requirements, step S5 is executed; the embodiment of the present invention can effectively improve detection efficiency and reduce costs by judging face quality to filter out low-quality image data to be detected.

[0097] S4, performing face recognition processing on the image data to be detected to obtain a second recognition result, confirming whether the second recognition result matches the face data in the face feature vector database, and when the second recognition result matches the face data in the face feature vector database, executing step S8, and when the second recognition result does not match the face data in the face feature vector database, executing step S5;

[0098] In step S4, the embodiment of the present invention performs face recognition processing on the image data to be detected, and obtaining a second recognition result includes: processing the image data to be detected based on a face recognition algorithm, extracting face feature data, and obtaining a second recognition result; confirming whether the face feature data matches the face data in the face feature vector database; when the face feature data matches the face data in the face feature vector database, executing step S8; when the face feature data does not match the face data in the face feature vector database, executing step S5, and storing the face feature data in the face feature vector database.

[0099] Specifically, the embodiment of the present invention extracts facial feature data based on a facial recognition algorithm (such as AdaFace, CosFace) through a trained facial recognition algorithm model, and sends the extracted facial feature data to a facial feature vector database (such as a faiss database) for facial matching, that is, the facial feature data is compared with the facial data in the facial feature vector database for cosine similarity. If the cosine similarity is greater than a first threshold (for example, 60% is expected), the match is considered successful and step S8 is executed. If the cosine similarity is greater than the first threshold, step S5 is executed, and the facial feature data is stored in the facial feature vector database for subsequent use.

[0100] S5, performing target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirming whether the third recognition result meets the target quality requirement, and executing step S6 when the third recognition result meets the target quality requirement; and discarding the image data to be detected and re-executing step S2 when the third recognition result does not meet the target quality requirement;

[0101] In step S5, the embodiment of the present invention performs target detection processing on the image data to be detected based on the multimodal module, and obtains a third recognition result including: performing target detection processing on the image data to be detected, identifying a target person area from the image data to be detected; intercepting image data of the target person area, confirming the target quality of the image data of the target person area, and obtaining a third recognition result.

[0102] The embodiment of the present invention performs target detection processing on the image data to be detected through a multimodal module to further judge the target quality of the target samples in the image data to be detected, thereby achieving secondary screening and further screening out low-quality image data to be detected, which can effectively improve detection efficiency and reduce costs.

[0103] S6, performing attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirming whether the fourth recognition result matches the attribute data in the attribute feature vector database, and when the fourth recognition result matches the attribute data in the attribute feature vector database, executing step S8, and when the fourth recognition result does not match the attribute data in the attribute feature vector database, executing step S7;

[0104] In step S6, the embodiment of the present invention performs attribute recognition processing on the image data to be detected based on the multimodal module, and obtains a fourth recognition result, including: performing attribute recognition processing on the image data to be detected based on the multimodal module, extracting attribute feature data, and obtaining a fourth recognition result; confirming whether the attribute feature data matches the attribute data in the attribute feature vector database; when the attribute feature data matches the attribute data in the attribute feature vector database, executing step S8; when the attribute feature data does not match the attribute data in the attribute feature vector database, executing step S7, and storing the attribute feature data in the attribute feature vector database.

[0105] The embodiment of the present invention performs attribute recognition processing on the image data to be detected through a multimodal module, and the attributes include but are not limited to front / back, gender, age, top color, top style, bottom color, bottom style, shoe color, shoe style, hat color, hat style, mask, hairstyle, glasses, etc. After extracting the corresponding attribute feature data, the attribute feature data is entered into an attribute feature vector database (for example, a faiss library) for attribute detection, that is, the attribute feature data is compared with the attribute data in the attribute feature vector database for cosine similarity. If it is greater than a second threshold (for example, it is expected to be 60%), it is considered that the match is successful, and step S8 is executed. If it is not greater than the second threshold, step S7 is executed, and the attribute feature data is stored in the attribute feature vector database for subsequent use.

[0106] S7, performing ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirming whether the fifth recognition result matches the ReID data in the ReID feature vector database, and when the fifth recognition result matches the ReID data in the ReID feature vector database, executing step S8;

[0107] In step S7, the embodiment of the present invention performs ReID detection processing on the image data to be detected based on the multimodal module, and obtains the fifth recognition result, including: performing ReID detection processing on the image data to be detected based on the multimodal module, extracting ReID feature data, and obtaining the fifth recognition result; confirming whether the ReID feature data matches the ReID data in the ReID feature vector database; when the ReID feature data matches the ReID data in the ReID feature vector database, executing step S8; when the ReID feature data does not match the ReID data in the ReID feature vector database, storing the ReID feature data in the ReID feature vector database.

[0108] The embodiment of the present invention performs ReID detection processing on the image data to be detected through a multimodal module, and compares the extracted ReID feature data with the ReID data in the ReID feature vector database (for example, the faiss library) for cosine similarity. If it is greater than a third threshold (for example, the expected value is 60%), the match is considered successful, and step S8 is executed. If it is not greater than the third threshold, the attribute feature data is stored in the attribute feature vector database for subsequent use.

[0109] S8. Output the pedestrian re-identification detection result.

[0110] In the embodiment of the present invention, the pedestrian re-identification detection result indicates that a pedestrian is successfully identified in the image data to be detected.

[0111] Compared with the prior art, the multimodal ReID detection method based on transfer learning described in the embodiment of the present invention has the following advantages:

[0112] 1. By reconstructing the data set, an original data set including visible light image data and infrared image data is obtained. A multimodal module is trained based on the original data set. When performing target detection, attribute recognition and pedestrian re-identification, even in the case of insufficient light such as night vision infrared and night, a high detection accuracy can be obtained, which has a high application promotion prospect;

[0113] 2. Visible light data and infrared data are mapped to the same feature space through transfer learning methods, which reduces the difference in feature distribution between visible light image data and infrared image data and improves detection accuracy.

[0114] Based on the above embodiment, as the above Figure 1 The present invention provides an embodiment of a multimodal ReID detection device based on transfer learning, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices, see Figure 3 As shown, the multimodal ReID detection device based on transfer learning includes:

[0115] The training module 100 is used to obtain an original data set based on the same scene, perform model training based on the original data set, and map multiple labeled data in the original data set to the same feature space through a transfer learning method to obtain a multimodal module through training;

[0116] The image analysis module 200 is used to obtain the video stream data to be detected in real time, analyze the video stream data, and obtain the image data to be detected of a single frame;

[0117] The face detection module 300 is used to perform face detection processing on the image data to be detected, obtain a first recognition result, confirm whether the first recognition result meets the face quality requirement, and output the image data to be detected to the face recognition module when the first recognition result meets the face quality requirement; when the first recognition result does not meet the face quality requirement, output the image data to be detected to the target detection module;

[0118] A face recognition module 400 is used to perform face recognition processing on the image data to be detected, obtain a second recognition result, confirm whether the second recognition result matches the face data in the face feature vector database, and output the image data to be detected to a result output module when the second recognition result matches the face data in the face feature vector database; when the second recognition result does not match the face data in the face feature vector database, output the image data to be detected to a target detection module;

[0119] The target detection module 500 is used to perform target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirm whether the third recognition result meets the target quality requirement, and output the image data to be detected to the attribute recognition module when the third recognition result meets the target quality requirement; when the third recognition result does not meet the target quality requirement, discard the image data to be detected and re-acquire the image data to be detected of the next frame;

[0120] The attribute recognition module 600 is used to perform attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirm whether the fourth recognition result matches the attribute data in the attribute feature vector database, and output the image data to be detected to the result output module when the fourth recognition result matches the attribute data in the attribute feature vector database; when the fourth recognition result does not match the attribute data in the attribute feature vector database, output the image data to be detected to the ReID detection module;

[0121] A ReID detection module 700 is used to perform ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirm whether the fifth recognition result matches the ReID data in the ReID feature vector database, and execute step S8 when the fifth recognition result matches the ReID data in the ReID feature vector database;

[0122] The result output module 800 is used to output the pedestrian re-identification detection result.

[0123] The multimodal ReID detection device based on transfer learning described in the embodiment of the present invention can execute the multimodal ReID detection method based on transfer learning provided in the above embodiment. The multimodal ReID detection device based on transfer learning has the corresponding functional steps and beneficial effects of the multimodal ReID detection method based on transfer learning described in the above embodiment. Please refer to the embodiment of the above-mentioned multimodal ReID detection method based on transfer learning for details. The embodiment of the present invention will not be repeated here.

[0124] An embodiment of the present invention also provides an electronic device, which may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the multimodal ReID detection method based on transfer learning in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, realizing the multimodal ReID detection method based on transfer learning in the above method embodiment.

[0125] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, a multimodal ReID detection method based on transfer learning as in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding relevant descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0126] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0127] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0128] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A multimodal ReID detection method based on transfer learning, characterized in that: The multimodal ReID detection method based on transfer learning includes: S1. Acquire an original data set based on the same scene, perform model training based on the original data set, map multiple labeled data in the original data set to the same feature space through a transfer learning method, and then train to obtain a multimodal module; S2, acquiring the video stream data to be detected in real time, parsing the video stream data, and acquiring the image data to be detected in a single frame; S3, performing face detection processing on the image data to be detected to obtain a first recognition result, confirming whether the first recognition result meets the face quality requirement, and executing step S4 when the first recognition result meets the face quality requirement, and executing step S5 when the first recognition result does not meet the face quality requirement; S4, performing face recognition processing on the image data to be detected to obtain a second recognition result, confirming whether the second recognition result matches the face data in the face feature vector database, and when the second recognition result matches the face data in the face feature vector database, executing step S8, and when the second recognition result does not match the face data in the face feature vector database, executing step S5; S5, performing target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirming whether the third recognition result meets the target quality requirement, and executing step S6 when the third recognition result meets the target quality requirement; and discarding the image data to be detected and re-executing step S2 when the third recognition result does not meet the target quality requirement; S6, performing attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirming whether the fourth recognition result matches the attribute data in the attribute feature vector database, and when the fourth recognition result matches the attribute data in the attribute feature vector database, executing step S8, and when the fourth recognition result does not match the attribute data in the attribute feature vector database, executing step S7; S7, performing ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirming whether the fifth recognition result matches the ReID data in the ReID feature vector database, and when the fifth recognition result matches the ReID data in the ReID feature vector database, executing step S8; S8. Output the pedestrian re-identification detection result.

2. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: In step S3, performing face detection and recognition processing on the image data to be detected to obtain a first recognition result includes: Processing the image data to be detected based on a face detection algorithm, and obtaining face quality parameters after detecting a face; Confirm whether the face quality parameters in the image data to be detected meet the face quality requirements, and obtain the first recognition result.

3. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: In step S4, performing face recognition processing on the image data to be detected to obtain a second recognition result includes: Processing the image data to be detected based on a face recognition algorithm, extracting face feature data, and obtaining a second recognition result; Confirming whether the facial feature data matches the facial data in a facial feature vector database; When the facial feature data matches the facial data in the facial feature vector database, executing step S8; When the facial feature data does not match the facial data in the facial feature vector database, step S5 is executed, and the facial feature data is stored in the facial feature vector database.

4. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: In step S5, the target detection processing is performed on the image data to be detected based on the multimodal module to obtain a third recognition result including: Performing target detection processing on the image data to be detected, and identifying a target person area from the image data to be detected; The image data of the target person area is intercepted, the target quality of the image data of the target person area is confirmed, and a third recognition result is obtained.

5. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: In step S6, the multimodal module is used to perform attribute recognition processing on the image data to be detected, and obtaining a fourth recognition result includes: Performing attribute recognition processing on the image data to be detected based on the multimodal module, extracting attribute feature data, and obtaining a fourth recognition result; confirming whether the attribute feature data matches the attribute data in the attribute feature vector database; When the attribute feature data matches the attribute data in the attribute feature vector database, executing step S8; When the attribute feature data does not match the attribute data in the attribute feature vector database, step S7 is executed, and the attribute feature data is stored in the attribute feature vector database.

6. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: In step S7, the ReID detection process is performed on the image data to be detected based on the multimodal module to obtain a fifth recognition result including: Performing ReID detection processing on the image data to be detected based on the multimodal module, extracting ReID feature data, and obtaining a fifth recognition result; Confirming whether the ReID feature data matches the ReID data in the ReID feature vector database; When the ReID feature data matches the ReID data in the ReID feature vector database, executing step S8; When the ReID feature data does not match the ReID data in the ReID feature vector database, the ReID feature data is stored in the ReID feature vector database.

7. The multimodal ReID detection method based on transfer learning according to claim 1, characterized in that: The original data set includes respectively acquired visible light image data of a first target sample, first infrared image data of the same source and second image data of a different source, and acquired third infrared image data of a second target sample.

8. A multimodal ReID detection device based on transfer learning, applied to the method according to any one of claims 1 to 7, characterized in that: The multimodal ReID detection device based on transfer learning includes: A training module is used to obtain an original data set based on the same scenario, perform model training based on the original data set, and map multiple labeled data in the original data set to the same feature space through a transfer learning method to obtain a multimodal module through training; An image analysis module is used to obtain the video stream data to be detected in real time, analyze the video stream data, and obtain the image data to be detected of a single frame; A face detection module is used to perform face detection processing on the image data to be detected, obtain a first recognition result, confirm whether the first recognition result meets the face quality requirement, and output the image data to be detected to the face recognition module when the first recognition result meets the face quality requirement; when the first recognition result does not meet the face quality requirement, output the image data to be detected to the target detection module; A face recognition module is used to perform face recognition processing on the image data to be detected to obtain a second recognition result, confirm whether the second recognition result matches the face data in the face feature vector database, and output the image data to be detected to a result output module when the second recognition result matches the face data in the face feature vector database; when the second recognition result does not match the face data in the face feature vector database, output the image data to be detected to a target detection module; The target detection module is used to perform target detection processing on the image data to be detected based on the multimodal module to obtain a third recognition result, confirm whether the third recognition result meets the target quality requirement, and output the image data to be detected to the attribute recognition module when the third recognition result meets the target quality requirement; when the third recognition result does not meet the target quality requirement, discard the image data to be detected and re-acquire the image data to be detected of the next frame; an attribute recognition module, configured to perform attribute recognition processing on the image data to be detected based on the multimodal module to obtain a fourth recognition result, confirm whether the fourth recognition result matches the attribute data in the attribute feature vector database, and output the image data to be detected to a result output module when the fourth recognition result matches the attribute data in the attribute feature vector database, and output the image data to be detected to a ReID detection module when the fourth recognition result does not match the attribute data in the attribute feature vector database; A ReID detection module is used to perform ReID detection processing on the image data to be detected based on the multimodal module to obtain a fifth recognition result, confirm whether the fifth recognition result matches the ReID data in the ReID feature vector database, and execute step S8 when the fifth recognition result matches the ReID data in the ReID feature vector database; The result output module is used to output the pedestrian re-identification detection results.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the processor implements instructions of any one of the methods described in claims 1-7.

Images