Image processing model training method and device, equipment and storage medium

Abstract

The invention relates to an image processing model training method and device, equipment and a storage medium. The method comprises the following steps: acquiring single-mode images obtained after different types of shooting devices shoot the same pedestrian; selecting a training sample from single-mode images obtained after shooting different pedestrians; inputting the training sample into an image processing model, and outputting a prediction classification result corresponding to the training sample; determining a loss function value based on the difference between the prediction classification result corresponding to the training sample and the real classification label of the training sample, and training the image processing model based on the loss function value; wherein a loss function correspondingly used by the loss function value is formed based on a double-end cross entropy loss function and a multi-mode negative loss function; until the training ending condition is achieved. By adopting the method, the training effect of cross-modal pedestrian re-identification can be improved, and the accuracy of a cross-modal pedestrian re-identification result is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a training method, apparatus, device and storage medium for an image processing model. Background Art

[0002] The road monitoring system is mainly used to maintain public security and prevent illegal and criminal incidents. The road monitoring system uses visible light cameras and infrared cameras to capture video images of pedestrians, and then identifies the same pedestrian targets through cross-camera video images, thereby determining their activity routes and trajectories. The monitoring cameras used in the road monitoring system include visible light cameras and infrared cameras.

[0003] Cross-modal person re-identification mainly studies the problem of trying to retrieve matching images belonging to the same pedestrian in the image library under two modalities given a red, green, and blue (RGB) image or infrared image of a specific pedestrian. When training cross-modal person re-identification, most existing methods assume that there are photos of each person captured by both visible light cameras and infrared cameras in the training set, and then use metric learning to pull the feature spaces of the same person under different types of cameras together to alleviate the modal differences between visible light and infrared cameras and provide supervision information for cross-modal matching. However, this assumption is not always true, because in the monitoring system, the camera can automatically switch to infrared mode when there is no light at night to capture identity information, and switch to visible light mode when there is light. Over such a long time span, it is impossible to guarantee that the same pedestrian will appear in the monitoring system at the same time.

[0004] Therefore, how to improve the training effect of cross-modal pedestrian re-identification and improve the accuracy of cross-modal pedestrian re-identification results remains an urgent problem to be solved. Summary of the invention

[0005] Based on this, it is necessary to provide a training method, device, equipment and storage medium for an image processing model that can improve the training effect of cross-modal pedestrian re-identification and improve the accuracy of cross-modal pedestrian re-identification results in response to the above-mentioned technical problems.

[0006] In a first aspect, a method for training an image processing model is provided, the method comprising:

[0007] Acquire a single-modal image obtained by photographing the same pedestrian using different types of photographing devices, where the types of photographing devices include at least a visible light camera and an infrared camera;

[0008] According to the number of pedestrians that the training samples need to cover, the training samples are selected from the single-modal images obtained after shooting different pedestrians;

[0009] Input the training samples into the image processing model, and output the prediction classification results corresponding to the training samples;

[0010] Based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, a loss function value is determined, and the image processing model is trained based on the loss function value; wherein the loss function used corresponding to the loss function value is based on a two-headed cross entropy loss function and a multimodal negative loss function, the two-headed cross entropy loss function is used to maximize the predicted classification result to be close to the true classification label, and the multimodal negative loss function is used to minimize the error of the predicted classification result;

[0011] If the training end condition is not met, return to the step of inputting the training sample into the image processing model, outputting the predicted classification result corresponding to the training sample and continue to execute until the training end condition is met.

[0012] In one embodiment, the loss function value includes a first loss function value and a second loss function value; and determining the loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample includes:

[0013] Get the probability that a single-modal image belongs to the true classification label;

[0014] After performing a smoothing operation on the true classification label, obtaining a smoothed true value of the true classification label;

[0015] In a case where the modes of the dual-head cross entropy loss function are derived from a visible light camera and an infrared camera, substituting the probability and the true value into the cross entropy loss function to obtain the first loss function value;

[0016] According to the predicted classification results, determining same-modality positive-difficult sample pairs, same-modality negative-difficult sample pairs, and cross-modality negative-difficult sample pairs;

[0017] Obtain an anchor feature. When the modalities of the multimodal negative loss function come from a visible light camera and an infrared camera, substitute the anchor feature, the same-modal positive-difficult sample pair, the same-modal negative-difficult sample pair, and the cross-modal negative-difficult sample pair into the multimodal negative loss function to obtain the second loss function value.

[0018] In one embodiment, substituting the anchor feature, the same-modality positive-difficult sample pair, the same-modality negative-difficult sample pair, and the cross-modality negative-difficult sample pair into the multimodal negative loss function to obtain the second loss function value includes:

[0019] Based on the multimodal negative loss function, determining a first distance metric between the anchor feature and the same-modal positive-difficult sample pair, a second distance metric between the anchor feature and the same-modal negative-difficult sample pair, and a third distance metric between the anchor feature and the cross-modal negative-difficult sample pair;

[0020] The second loss function value is obtained according to a first difference between the first distance metric and the third distance metric and a second difference between the third distance metric and the second distance metric.

[0021] In one embodiment, obtaining the second loss function value according to a first difference between the first distance metric and the third distance metric and a second difference between the third distance metric and the second distance metric includes:

[0022] Substituting the first difference into a first constraint function of the multimodal negative loss function to obtain a first constraint value; wherein the first constraint function is used to make the first difference greater than or equal to 0;

[0023] Substituting the second difference into the second constraint function of the multimodal negative loss function to obtain a second constraint value; wherein the second constraint function is used to make the second difference greater than or equal to 0;

[0024] The second loss function value is determined according to the first constraint value and the second constraint value.

[0025] In one of the embodiments, the constraint condition of the multimodal negative loss function is: the first distance metric is smaller than the third distance metric, and the third distance metric is smaller than the second distance metric.

[0026] In one embodiment, the method further comprises:

[0027] Obtain the target image processing model obtained after training;

[0028] Multiple pedestrian images to be identified are input into the target image processing model to obtain the predicted classification result of each pedestrian image, and pedestrians with the same predicted classification result are regarded as the same pedestrian; wherein the multiple pedestrian images are obtained by photographing the pedestrians with different types of photographing equipment.

[0029] In a second aspect, the present application provides a training device for an image processing model, the device comprising:

[0030] An acquisition module, used to acquire a single-mode image obtained by photographing the same pedestrian using different types of photographing devices, wherein the types of photographing devices include at least a visible light camera and an infrared camera;

[0031] The acquisition module is further used to select training samples from the single-modal images obtained by photographing different pedestrians according to the number of pedestrians that the training samples need to cover;

[0032] A processing module, used for inputting the training samples into an image processing model and outputting the prediction classification results corresponding to the training samples;

[0033] A training module, used to determine a loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, and train the image processing model based on the loss function value; wherein the loss function used corresponding to the loss function value is based on a two-headed cross entropy loss function and a multimodal negative loss function, the two-headed cross entropy loss function is used to maximize the predicted classification result close to the true classification label, and the multimodal negative loss function is used to minimize the error of the predicted classification result;

[0034] The training module is also used to return to the step of inputting the training sample into the image processing model, outputting the corresponding predicted classification result of the training sample and continuing to execute until the training end condition is met.

[0035] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0036] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0037] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0038] The training method, apparatus, device and storage medium of the image processing model provided in the present application aim to train the image processing model to achieve cross-modal pedestrian matching in the absence of cross-modal pedestrian data pairs, and to achieve this goal by proposing a two-headed cross entropy loss function and a multi-modal negative loss function. Traditional cross entropy loss and metric learning loss push negative samples away from anchor samples at the same time without considering modal information. However, pushing away cross-modal negative samples will further widen the cross-modal gap and lead to mismatches. This embodiment proposes a two-headed cross entropy loss function and a multi-modal negative loss function, which can alleviate cross-modal contrast and enhance intra-modal contrast. The use of a two-headed cross entropy loss function and a multi-modal negative loss function in the training of the image processing model can enable the trained target processing model to solve the cross-modal pedestrian matching problem in the absence of cross-modal pedestrian data pairs, achieving a technical effect of high pedestrian matching accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A diagram of an application environment of a training method for an image processing model in one embodiment;

[0041] Figure 2 A schematic diagram of a flow chart of a method for training an image processing model in one embodiment;

[0042] Figure 3 A partial flow chart of a method for training an image processing model in one embodiment;

[0043] Figure 4 is a structural block diagram of a training device for an image processing model in one embodiment;

[0044] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The training method of the image processing model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0047] In an exemplary embodiment, Figure 2 As shown, a training method for an image processing model is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 204. Among them:

[0048] Step 201 , obtain unimodal images obtained by photographing the same pedestrian using different types of photographing equipment, and select training samples from the unimodal images obtained by photographing different pedestrians according to the number of pedestrians that the training samples need to cover.

[0049] The types of the photographing device include at least a visible light camera and an infrared camera, and may also include other types of photographing devices, which are not limited in this embodiment. For the convenience of description, the following describes the training method of the image processing model based on the assumption that the types of the photographing device include a visible light camera and an infrared camera.

[0050] Among them, a unimodal image refers to an image acquired using the same imaging device or technology, and its image features and data distribution are relatively simple, usually showing an obvious unimodal characteristic, that is, the data is concentrated in a small range at or near one. The unimodal image described in this embodiment refers to an image acquired using the same type of shooting device, for example, an image acquired using a visible light camera is a unimodal image, and an image acquired using an infrared camera is a unimodal image. The types of shooting devices include visible light cameras and infrared cameras, and the acquired unimodal images include unimodal images taken by visible light cameras, and also include unimodal images taken by infrared cameras.

[0051] The number of pedestrians to be covered can be set according to actual needs and is not limited in this embodiment.

[0052] It should be noted that one pedestrian described in this embodiment represents one identity.

[0053] The training sample can be expressed as .in , Represents visible light camera, Represents infrared camera, Represents the total number of visible light cameras, Represents the total number of infrared cameras. The training samples are divided into training set and test set. The unimodal images contained in the training set do not have classification results (which can also be understood as classification labels or identity labels). The images contained in the test set have real classification results (which can also be understood as real classification labels or real identity labels).

[0054] It is understandable that for each camera A collection of unimodal images under In terms of , the set of unimodal images with classification labels is represented as , represents a photo of a pedestrian, represents the corresponding classification label (i.e. identity label), is the number of single-mode images captured by the image camera. If the image is a single-mode image captured by the same type of camera, then for the camera and camera ,when or Time Satisfaction However, since the visible light camera is used during the day and the infrared camera is used at night, the time span of camera switching is long, so there is no potential cross-modal positive sample pair in the training samples. That is, for the visible light camera and the infrared camera, when , ,exist The purpose of training the following image processing model is to match pedestrians with the same predicted classification result (which can also be understood as predicted classification label or predicted identity label) from images of different modalities, and regard pedestrians with the same predicted classification result as the same pedestrian.

[0055] Optionally, training samples can be obtained from a pedestrian re-identification dataset (such as the SYSU-MM01 dataset). Specifically, the number of pedestrians to be covered is set to 395 pedestrians, and there are images of each pedestrian under 6 cameras. The 6 cameras include 4 visible light cameras and 2 infrared cameras. From the 395 monomodal images of pedestrians, 197 monomodal images of pedestrians under visible light cameras and 198 monomodal images of pedestrians under infrared cameras are randomly selected. The 197 monomodal images of pedestrians under visible light cameras and the 198 monomodal images of pedestrians under infrared cameras are used as training samples, which contain 11,384 monomodal images corresponding to visible light cameras and 5,874 monomodal images corresponding to infrared cameras, totaling 17,258 images.

[0056] Step 202: input the training sample into the image processing model, and output the prediction classification result corresponding to the training sample.

[0057] The image processing model consists of a feature extractor and a two-headed classifier Composition. Feature Extractor Take a single modality image Mapped to a feature vector , each classifier further transforms the feature vector of the specific mode As input and output classification logic , here is the number of categories for each modality. Used to output the corresponding predicted classification results of the training sample.

[0058] The predicted classification result can also be understood as a predicted classification label or a predicted identity label.

[0059] In one embodiment, the image processing model is, for example, a convolutional neural network, and a residual modal elimination network (such as a ResNet-50 network) pre-trained on ImageNet is used as the backbone of the two branches. The global average pooling (GAP) layer of the convolutional neural network is replaced by generalized average pooling. Since shape representation does not involve modal difference issues, the residual modal elimination network is only used in branches that take the original image as input. The data enhancement strategy follows the implementation in fast-reid. The size of the mini-batch is 128. During the training process, Adam is used as the optimizer, the initial learning rate is set to 0.0006, and it decays by 0.1 after the 30th and 60th iterations, for a total of 100 iterations. Model training is performed after adding double-headed cross entropy loss and multimodal negative loss to the loss function.

[0060] Step 203: Determine a loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, and train the image processing model based on the loss function value.

[0061] The loss function used in the loss function value is based on the two-headed cross entropy loss function and the multimodal negative loss function. The two-headed cross entropy loss function is used to maximize the predicted classification result close to the true classification label. The multimodal negative loss function is used to minimize the error of the predicted classification result.

[0062] In the training process of the original image processing model, the loss function used by the classifier is the cross-entropy loss function. When minimizing the cross-entropy loss, each feature image is forced to be close to its corresponding class prototype and away from other class prototypes. In other words, the traditional cross-entropy loss function pushes cross-modal features apart, which further exacerbates the modal differences and leads to poor cross-modal matching capabilities.

[0063] To alleviate this problem, this embodiment adds a two-headed cross entropy loss function during the training process of the image processing model.

[0064] The two-headed cross entropy loss function can be expressed as Where Represents a single-modality image The probability of belonging to the true classification label, Represents the true value of the true classification label after smoothing (which can also be understood as smoothing initialization). Indicates that the modality comes from the visible light camera and the infrared camera. Indicates the number of categories for each modality.

[0065] The training of traditional image processing models uses metric learning loss as a supplement to the classification loss for discriminative feature learning, which is often used in person re-identification tasks to optimize local relationships at the representation level. However, when dealing with the problem of cross-modal missing person re-identification, traditional metric learning losses (such as triplet loss) are not credible because when cross-modal positive sample pairs are missing, traditional metric learning losses will only cluster intra-modal samples together and push cross-modal samples apart, which further exacerbates the modality difference and leads to performance degradation.

[0066] To overcome this problem, this embodiment proposes a multimodal negative loss function. Given an anchor feature , we simultaneously sample the modal positive hard sample pairs , the negative hard sample pair within the modality and cross-modal negative hard sample pairs . Let anchor feature and its modal intrinsic positive difficult sample pair The distance is less than the anchor feature and its cross-modal negative difficult sample pairs At the same time, we let the anchor feature and its cross-modal negative difficult sample pairs The distance is less than the anchor feature and its intra-modal negative hard sample pairs The distance between the two modalities is used to penalize the modal differences.

[0067] Among them, positive-hard sample pairs are an important concept in machine learning and deep learning, especially in the fields of object detection and face recognition. Positive-hard sample pairs refer to sample pairs that are both positive samples and hard samples. These sample pairs are crucial for model training and optimization because they can reveal the weaknesses of the model and help the model learn more robust and generalized features. Among them, in tasks such as object detection, positive samples usually refer to samples that match the position of the real target (such as objects, faces, etc.). That is, samples whose intersection-over-union ratio of the algorithm detection box and the real target box is greater than a preset threshold are considered positive samples. Hard samples refer to samples that are difficult for the model to correctly classify or identify. In object detection, hard samples may include those with severe occlusion, poor lighting conditions, large angle changes, or unclear features of the target itself. For positive samples, if the model has a low probability of predicting them, then these positive samples can be considered hard samples.

[0068] Among them, negative hard samples refer to those sample pairs that are both negative samples and hard samples. In binary or multi-classification problems, negative samples refer to those samples that do not belong to the category we are interested in. For example, in the image classification task of identifying cats and dogs, if our goal is to identify cats, then all dog images are negative samples. Hard samples usually refer to those samples that are difficult for the model to correctly classify or predict. For negative samples, if the model predicts a low probability for them, then these negative samples can be regarded as hard samples.

[0069] Among them, cross-modal negative difficult sample pairs refer to a pair of negative samples from different modalities in cross-modal retrieval or classification tasks. These samples are difficult to distinguish for the model, that is, the model may mistakenly think that they are positive sample pairs (i.e., similar or related).

[0070] The multimodal negative loss function can be expressed as:

[0071]

[0072] Here represents the distance metric, and is a margin with a default value of 0.1. Compared with the traditional triplet loss, the multimodal negative loss ensures discriminative representation learning by pulling cross-modal negative hard sample pairs together, while alleviating modality differences by pushing intra-modal negative hard sample pairs far apart, which is beneficial for cross-modal matching.

[0073] Step 204, when the training end condition is not met, return to the step of inputting the training sample into the image processing model, outputting the predicted classification result corresponding to the training sample and continue to execute until the training end condition is met.

[0074] The training end condition may be the number of training times, the training end condition may also include the training duration or other conditions, which are not limited in this embodiment.

[0075] When the training end condition is not met, step 202 to step 203 need to be repeatedly performed until the training end condition is met.

[0076] Optionally, after each training, whether the prediction classification result output by the trained image processing model is accurate is evaluated based on the test set in the training sample. If it is inaccurate, training is performed again.

[0077] After the training end condition is reached, the target image processing model can be obtained.

[0078] The training method of the image processing model provided in this embodiment aims to train the image processing model to achieve cross-modal pedestrian matching in the absence of cross-modal pedestrian data pairs, and achieves this goal by proposing a two-headed cross entropy loss function and a multi-modal negative loss function. Traditional cross entropy loss and metric learning loss push negative samples away from anchor samples at the same time without considering modal information. However, pushing away cross-modal negative samples will further widen the cross-modal gap and lead to mismatches. This embodiment proposes a two-headed cross entropy loss function and a multi-modal negative loss function, which can alleviate cross-modal contrast and enhance intra-modal contrast. The use of a two-headed cross entropy loss function and a multi-modal negative loss function in the training of the image processing model can enable the trained target processing model to solve the cross-modal pedestrian matching problem in the absence of cross-modal pedestrian data pairs, achieving a technical effect of high pedestrian matching accuracy.

[0079] In an exemplary embodiment, the loss function value includes a first loss function value and a second loss function value.

[0080] like Figure 3 As shown, step 203 determines the loss function value based on the difference between the predicted classification result corresponding to the training sample and the real classification label of the training sample, including steps 301 to 305. Among them:

[0081] Step 301, obtaining the probability that the unimodal image belongs to the true classification label.

[0082] The probability that a unimodal image belongs to the true classification label is output by the two-head classifier in this image processing model.

[0083] Step 302: After performing a smoothing operation on the real classification label, obtain the real value of the smoothed real classification label.

[0084] Label smoothing is a regularization method that is often used in deep learning models for classification problems to prevent the model from being overconfident about certain labels in the training data, which can lead to overfitting.

[0085] In the traditional training process, the predicted classification label of each training sample is usually represented by a one-hot encoding vector, where the correct category position is 1 and the rest are 0. However, label smoothing techniques modify these hard target labels and give categories other than the correct category a smaller positive value, thereby making the distribution of the true classification label slightly "smooth".

[0086] Step 303, when the modes of the dual-head cross entropy loss function are derived from the visible light camera and the infrared camera, substitute the probability and the true value into the cross entropy loss function to obtain the first loss function value.

[0087] The two-headed cross entropy loss function can be expressed as Where Represents a single-modality image The probability of belonging to the true classification label, Represents the true value of the true classification label after smoothing (which can also be understood as smoothing initialization). Indicates that the modality comes from the visible light camera and the infrared camera. Indicates the number of categories for each modality.

[0088] Substitute the probability and the true value into the cross entropy loss function, that is, let is equal to the probability obtained in step 301, and is equal to the true value obtained in step 302, we can get The value of is the first loss function value.

[0089] Step 304: Determine same-modality positive and difficult sample pairs, same-modality negative and difficult sample pairs, and cross-modality negative and difficult sample pairs according to the predicted classification result.

[0090] Among them, positive-hard sample pairs are an important concept in machine learning and deep learning, especially in the fields of object detection and face recognition. Positive-hard sample pairs refer to sample pairs that are both positive samples and hard samples. These sample pairs are crucial for model training and optimization because they can reveal the weaknesses of the model and help the model learn more robust and generalized features. Among them, in tasks such as object detection, positive samples usually refer to samples that match the position of the real target (such as objects, faces, etc.). That is, samples whose intersection-over-union ratio of the algorithm detection box and the real target box is greater than a preset threshold are considered positive samples. Hard samples refer to samples that are difficult for the model to correctly classify or identify. In object detection, hard samples may include those with severe occlusion, poor lighting conditions, large angle changes, or unclear features of the target itself. For positive samples, if the model has a low probability of predicting them, then these positive samples can be considered hard samples.

[0091] Among them, negative hard samples refer to those sample pairs that are both negative samples and hard samples. In binary or multi-classification problems, negative samples refer to those samples that do not belong to the category we are interested in. For example, in the image classification task of identifying cats and dogs, if our goal is to identify cats, then all dog images are negative samples. Hard samples usually refer to those samples that are difficult for the model to correctly classify or predict. For negative samples, if the model predicts a low probability for them, then these negative samples can be regarded as hard samples.

[0092] Among them, cross-modal negative difficult sample pairs refer to a pair of negative samples from different modalities in cross-modal retrieval or classification tasks. These samples are difficult to distinguish for the model, that is, the model may mistakenly think that they are positive sample pairs (i.e., similar or related).

[0093] Step 305, obtain the anchor feature. When the modalities of the multimodal negative loss function come from the visible light camera and the infrared camera, substitute the anchor feature, the same-modal positive-difficult sample pair, the same-modal negative-difficult sample pair and the cross-modal negative-difficult sample pair into the multimodal negative loss function to obtain the second loss function value.

[0094] The multimodal negative loss function can be expressed as:

[0095]

[0096] Here represents the distance metric, and The default value is set to 0.1. Indicates that the modality comes from the visible light camera and the infrared camera. N indicates the number of single-modality images.

[0097] Based on the multimodal negative loss function, a first distance metric between the anchor feature and the same-modal positive and difficult sample pair, a second distance metric between the anchor feature and the same-modal negative and difficult sample pair, and a third distance metric between the anchor feature and the cross-modal negative and difficult sample pair can be determined. The first distance metric is expressed as The second distance metric is expressed as The third distance metric is expressed as . A first difference between the first distance metric and the third distance metric can also be obtained, and the first difference is expressed as A second difference between the third distance metric and the second distance metric can also be obtained, and the second difference is expressed as The second loss function value is obtained according to the first difference between the first distance metric and the third distance metric and the second difference between the third distance metric and the second distance metric. .

[0098] More specifically, the first difference is substituted into the first constraint function of the multimodal negative loss function to obtain a first constraint value. The second difference is substituted into the second constraint function of the multimodal negative loss function to obtain a second constraint value. The second loss function value is determined based on the first constraint value and the second constraint value.

[0099] The meaning of the first constraint function is , which is used to make the minimum value 0. The first constraint function in the multimodal negative loss function is expressed as , used to make the first difference greater than or equal to 0. The meaning of the second constraint function is , which is used to make the minimum value 0. The second constraint function in the multimodal negative loss function is expressed as , used to make the second difference greater than or equal to 0.

[0100] It should be noted that the constraints of the multimodal negative loss function are: the first distance metric is less than the third distance metric, and the third distance metric is less than the second distance metric. and its cross-modal negative difficult sample pairs The distance is less than the anchor feature and its intra-modal negative hard sample pairs The distance between the two modalities is used to penalize the modal differences.

[0101] The method provided in this embodiment proposes a two-headed cross entropy loss function and a multimodal negative loss function, which can alleviate cross-modal contrast and enhance intra-modal contrast. The use of a two-headed cross entropy loss function and a multimodal negative loss function in the training of an image processing model can enable the trained target processing model to solve the cross-modal pedestrian matching problem in the absence of cross-modal pedestrian data pairs, thereby achieving a technical effect of high pedestrian matching accuracy.

[0102] In some embodiments, the training method of the image processing model further includes:

[0103] Step 1: Obtain the target image processing model after training.

[0104] When the training end condition is met, the image processing model obtained by the last training is obtained as the target image processing model.

[0105] Step 2: Input multiple pedestrian images to be identified into the target image processing model to obtain the predicted classification result of each pedestrian image, and regard pedestrians with the same predicted classification result as the same pedestrian.

[0106] The multiple pedestrian images are obtained by photographing pedestrians using different types of photographing equipment, which at least include visible light cameras and infrared cameras.

[0107] The target image processing model is capable of identifying multiple pedestrian images across modalities and outputting a predicted classification result for each pedestrian image, wherein the predicted classification result includes a predicted classification label.

[0108] The method provided in this embodiment can enable the trained target processing model to solve the cross-modal pedestrian matching problem in the absence of cross-modal pedestrian data pairs, thereby achieving a technical effect of high pedestrian matching accuracy.

[0109] It should be understood that, although the steps in the flowcharts of the embodiments described above are shown in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0110] Based on the same inventive concept, the embodiment of the present application also provides a training device for an image processing model for implementing the training method of the image processing model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more image processing models provided below can refer to the limitations of the training method for the image processing model above, and will not be repeated here.

[0111] In an exemplary embodiment, Figure 4 As shown, a training device 40 for an image processing model is provided, comprising: an acquisition module 41, a processing module 42 and a training module 43, wherein:

[0112] The acquisition module 41 is used to acquire a single-mode image obtained after the same pedestrian is photographed by different types of photographing devices, and the types of the photographing devices include at least a visible light camera and an infrared camera.

[0113] The acquisition module 41 is further used to select training samples from the single-modal images obtained by photographing different pedestrians according to the number of pedestrians that the training samples need to cover.

[0114] The processing module 42 is used to input the training sample into the image processing model and output the prediction classification result corresponding to the training sample.

[0115] The training module 43 is used to determine the loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, and train the image processing model based on the loss function value; wherein the loss function used corresponding to the loss function value is based on a two-headed cross entropy loss function and a multimodal negative loss function, the two-headed cross entropy loss function is used to maximize the predicted classification result to be close to the true classification label, and the multimodal negative loss function is used to minimize the error of the predicted classification result.

[0116] The training module 43 is also used to return to the step of inputting the training sample into the image processing model, outputting the predicted classification result corresponding to the training sample and continuing to execute until the training end condition is met.

[0117] In some embodiments, the loss function value includes a first loss function value and a second loss function value; the training module 43 is used to: obtain the probability that a unimodal image belongs to a true classification label; after smoothing the true classification label, obtain the true value of the smoothed true classification label; when the modality of the dual-headed cross entropy loss function comes from a visible light camera and an infrared camera, substitute the probability and the true value into the cross entropy loss function to obtain the first loss function value; determine the same-modal positive difficult sample pairs, the same-modal negative difficult sample pairs and the cross-modal negative difficult sample pairs according to the predicted classification results; obtain the anchor feature, and when the modality of the multimodal negative loss function comes from a visible light camera and an infrared camera, substitute the anchor feature, the same-modal positive difficult sample pairs, the same-modal negative difficult sample pairs and the cross-modal negative difficult sample pairs into the multimodal negative loss function to obtain the second loss function value.

[0118] In some embodiments, the training module 43 is used to: determine, based on the multimodal negative loss function, a first distance metric between the anchor feature and the homomodal positive-difficult sample pair, a second distance metric between the anchor feature and the homomodal negative-difficult sample pair, and a third distance metric between the anchor feature and the cross-modal negative-difficult sample pair; and obtain the second loss function value according to a first difference between the first distance metric and the third distance metric and a second difference between the third distance metric and the second distance metric.

[0119] In some embodiments, the training module 43 is used to: substitute the first difference into the first constraint function of the multimodal negative loss function to obtain a first constraint value; wherein the first constraint function is used to make the first difference greater than or equal to 0; substitute the second difference into the second constraint function of the multimodal negative loss function to obtain a second constraint value; wherein the second constraint function is used to make the second difference greater than or equal to 0; determine the second loss function value based on the first constraint value and the second constraint value.

[0120] In some embodiments, the constraint condition of the multimodal negative loss function is that the first distance metric is smaller than the third distance metric, and the third distance metric is smaller than the second distance metric.

[0121] In some embodiments, the processing module 42 is also used to: obtain a target image processing model obtained after training; input multiple pedestrian images to be identified into the target image processing model to obtain a predicted classification result for each pedestrian image, and treat pedestrians with the same predicted classification result as the same pedestrian; wherein the multiple pedestrian images are obtained after the pedestrians are photographed by different types of shooting devices.

[0122] Each module in the training device 40 of the image processing module can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as training samples and predicted classification results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the training method of the image processing module provided in any of the above embodiments is implemented.

[0124] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the training method of the image processing module provided in any of the above embodiments is implemented.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the training method of the image processing model provided in any of the above embodiments is implemented.

[0127] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements a training method for an image processing model as provided in any of the above embodiments.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0131] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for an image processing model, characterized in that: The method comprises: Acquire a single-modal image obtained by photographing the same pedestrian using different types of photographing devices, where the types of photographing devices include at least a visible light camera and an infrared camera; According to the number of pedestrians that the training samples need to cover, the training samples are selected from the single-modal images obtained after shooting different pedestrians; Input the training samples into the image processing model, and output the prediction classification results corresponding to the training samples; Based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, a loss function value is determined, and the image processing model is trained based on the loss function value; wherein the loss function used corresponding to the loss function value is based on a two-headed cross entropy loss function and a multimodal negative loss function, the two-headed cross entropy loss function is used to maximize the predicted classification result to be close to the true classification label, and the multimodal negative loss function is used to minimize the error of the predicted classification result; If the training end condition is not met, return to the step of inputting the training sample into the image processing model, outputting the predicted classification result corresponding to the training sample and continue to execute until the training end condition is met.

2. The method according to claim 1, characterized in that The loss function value includes a first loss function value and a second loss function value; and determining the loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample includes: Get the probability that a single-modal image belongs to the true classification label; After performing a smoothing operation on the true classification label, obtaining a smoothed true value of the true classification label; In a case where the modes of the dual-head cross entropy loss function are derived from a visible light camera and an infrared camera, substituting the probability and the true value into the cross entropy loss function to obtain the first loss function value; According to the predicted classification results, determining same-modality positive-difficult sample pairs, same-modality negative-difficult sample pairs, and cross-modality negative-difficult sample pairs; Obtain an anchor feature. When the modalities of the multimodal negative loss function come from a visible light camera and an infrared camera, substitute the anchor feature, the same-modal positive-difficult sample pair, the same-modal negative-difficult sample pair, and the cross-modal negative-difficult sample pair into the multimodal negative loss function to obtain the second loss function value.

3. The method according to claim 2, characterized in that Substituting the anchor feature, the same-modality positive-difficult sample pair, the same-modality negative-difficult sample pair, and the cross-modality negative-difficult sample pair into the multimodal negative loss function to obtain the second loss function value includes: Based on the multimodal negative loss function, determining a first distance metric between the anchor feature and the same-modal positive-difficult sample pair, a second distance metric between the anchor feature and the same-modal negative-difficult sample pair, and a third distance metric between the anchor feature and the cross-modal negative-difficult sample pair; The second loss function value is obtained according to a first difference between the first distance metric and the third distance metric and a second difference between the third distance metric and the second distance metric.

4. The method according to claim 3, characterized in that The obtaining the second loss function value according to a first difference between the first distance metric and the third distance metric and a second difference between the third distance metric and the second distance metric comprises: Substituting the first difference into a first constraint function of the multimodal negative loss function to obtain a first constraint value; wherein the first constraint function is used to make the first difference greater than or equal to 0; Substituting the second difference into the second constraint function of the multimodal negative loss function to obtain a second constraint value; wherein the second constraint function is used to make the second difference greater than or equal to 0; The second loss function value is determined according to the first constraint value and the second constraint value.

5. The method according to claim 3, characterized in that: The constraints of the multimodal negative loss function are: the first distance metric is smaller than the third distance metric, and the third distance metric is smaller than the second distance metric.

6. The method according to claim 1, characterized in that The method further comprises: Obtain the target image processing model obtained after training; Multiple pedestrian images to be identified are input into the target image processing model to obtain the predicted classification result of each pedestrian image, and pedestrians with the same predicted classification result are regarded as the same pedestrian; wherein the multiple pedestrian images are obtained by photographing the pedestrians with different types of photographing equipment.

7. A training device for an image processing model, characterized in that: The device comprises: An acquisition module, used to acquire a single-mode image obtained by photographing the same pedestrian using different types of photographing devices, wherein the types of photographing devices include at least a visible light camera and an infrared camera; The acquisition module is further used to select training samples from the single-modal images obtained after shooting different pedestrians according to the number of pedestrians that the training samples need to cover; A processing module, used for inputting the training samples into an image processing model and outputting the prediction classification results corresponding to the training samples; A training module, used to determine a loss function value based on the difference between the predicted classification result corresponding to the training sample and the true classification label of the training sample, and train the image processing model based on the loss function value; wherein the loss function used corresponding to the loss function value is based on a two-headed cross entropy loss function and a multimodal negative loss function, the two-headed cross entropy loss function is used to maximize the predicted classification result close to the true classification label, and the multimodal negative loss function is used to minimize the error of the predicted classification result; The training module is also used to return to the step of inputting the training sample into the image processing model, outputting the corresponding predicted classification result of the training sample and continuing to execute until the training end condition is met.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images