Model training, image processing method and device, and electronic equipment

Abstract

The present disclosure provides a model training, image processing method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, especially to the technical field of deep learning, image processing, computer vision and the like. The specific implementation scheme is: inputting a sample image into a preset model to obtain a feature map of the sample image output by a first intermediate layer of the preset model; inputting an image in each bounding box of one or more bounding boxes contained in the sample image into the preset model respectively to obtain a feature vector corresponding to each bounding box output by a second intermediate layer of the preset model; generating a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box; updating the preset model based on the target loss function, and obtaining a target model in the case that the preset model meets a convergence condition. The above method improves the efficiency of model training and ensures accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, image processing, and computer vision. Background Technology

[0002] With the development of computer technology, visual target tracking models are increasingly widely used in the field of computer vision to track visual targets. However, the training of visual target tracking models is supervised training, which requires extensive manual annotation of each frame in a series of video frames as training data. The model is then trained based on these annotated video frames. Because a large number of video frames need to be processed to complete the training, the efficiency is relatively low. Therefore, how to improve training efficiency while ensuring the accuracy of the model during training has become a problem that needs to be solved. Summary of the Invention

[0003] This disclosure provides a model training method, image processing method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, a model training method is provided, comprising:

[0005] A sample image is input into a preset model to obtain a feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image.

[0006] The image within each of one or more bounding boxes contained in the sample image is input into a preset model to obtain the feature vector corresponding to each bounding box output by the second intermediate layer of the preset model;

[0007] Based on the feature map of the sample image and the feature vector corresponding to each bounding box, a target loss function is generated.

[0008] The preset model is updated based on the target loss function. When the preset model meets the convergence condition, a target model is obtained. The target model is used to process the target image to obtain a feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching.

[0009] According to a second aspect of this disclosure, an image processing method is provided, comprising:

[0010] The target image is input into the target model to obtain the feature map of the target image output by the target model. The feature map of the target image is smaller than the target image. The feature map of the target image is used for tracking and matching. The target model is obtained based on the model training method of the first aspect.

[0011] According to a third aspect of this disclosure, a model training apparatus is provided, comprising:

[0012] The feature extraction module is used to input a sample image into a preset model to obtain a feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image; and to input the image within each of one or more bounding boxes contained in the sample image into the preset model to obtain a feature vector corresponding to each bounding box output by the second intermediate layer of the preset model.

[0013] The target loss function generation module is used to generate a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box;

[0014] The target model update module is used to update the preset model based on the target loss function. When the preset model meets the convergence condition, a target model is obtained. The target model is used to process the target image to obtain a feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching.

[0015] According to a fourth aspect of this disclosure, an image processing apparatus is provided, comprising:

[0016] An image processing module is used to input a target image into a target model to obtain a feature map of the target image output by the target model. The feature map of the target image is smaller than the target image. The feature map of the target image is used for tracking and matching. The target model is obtained by the model training method of the first method.

[0017] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0018] At least one processor; and

[0019] The memory is communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the map generation method of the first aspect described above.

[0021] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the aforementioned method.

[0022] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned method.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0024] The solution provided in this embodiment only requires inputting a sample image into a preset model to obtain a feature map of the sample image output by the preset model. Then, the images within each bounding box of the sample image are input into the preset model to obtain a feature vector corresponding to each bounding box. Based on the feature map of the sample image and the feature vector corresponding to each bounding box, a loss function is generated, and the preset model is updated to obtain the target model. Thus, training can be completed only with labeled bounding boxes in the sample image, avoiding the low efficiency problem caused by labeling consecutive video frames in existing technologies, while ensuring accuracy. Furthermore, since the loss function is constructed using smaller-sized feature maps and feature vectors output from intermediate layers during model training, the problem of high computational cost associated with constructing the loss function using larger-sized final output images can be avoided, improving the efficiency of model training and ensuring accuracy. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a schematic flowchart of a model training method according to an embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram of the composition structure of a preset model according to an embodiment of the present disclosure;

[0028] Figure 3 This is a schematic diagram of the feature vector corresponding to a bounding box according to an embodiment of the present disclosure;

[0029] Figure 4 This is a schematic flowchart of an image processing method according to an embodiment of the present disclosure;

[0030] Figure 5 This is an exemplary flowchart illustrating a processing scenario of an image processing method according to an embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram of the composition structure of a model training device according to an embodiment of the present disclosure;

[0032] Figure 7 This is a schematic diagram of the composition structure of an image processing apparatus according to an embodiment of the present disclosure;

[0033] Figure 8 This is a schematic diagram of the composition structure of an image processing apparatus according to another embodiment of the present disclosure;

[0034] Figure 9 This is a block diagram of an electronic device used to implement the model training method and image processing method of the embodiments of this disclosure. Detailed Implementation

[0035] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] The first aspect of this disclosure provides a model training method, such as... Figure 1 As shown, it includes:

[0037] S101: Input the sample image into the preset model to obtain the feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image;

[0038] S102: Input the image within each of one or more bounding boxes contained in the sample image into a preset model to obtain the feature vector corresponding to each bounding box output by the second intermediate layer of the preset model;

[0039] S103: Generate a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box;

[0040] S104: Update the preset model based on the target loss function. If the preset model meets the convergence condition, obtain the target model. The target model is used to process the target image to obtain the feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching.

[0041] The above model training method can be implemented using electronic devices. These electronic devices may include, but are not limited to, one of the following: servers, desktop computers, tablet computers, etc.

[0042] It should be noted that the processing order of the above S101 and S102 can be either S101 is executed first and then S102, or S102 is executed first and then S101. As long as S101 and S102 are completed before S103 is executed, they are within the protection scope of this embodiment.

[0043] By adopting the above scheme, only the sample image needs to be input into a preset model to obtain the feature map of the sample image output by the preset model. The images within each bounding box of the sample image are then input into the preset model to obtain the feature vector corresponding to each bounding box. Based on the feature map of the sample image and the feature vector corresponding to each bounding box, a loss function is generated, and the preset model is then updated to obtain the target model. Thus, training can be completed only with labeled bounding boxes in the sample image, avoiding the low efficiency problem caused by labeling consecutive video frames in existing technologies, while ensuring accuracy. Furthermore, since the loss function is constructed using the smaller-sized feature map and feature vector output from the intermediate layer during model training, the problem of high computational cost associated with constructing the loss function using the larger-sized final output image can be avoided, improving the efficiency of model training and ensuring accuracy.

[0044] The sample image described above may be an image labeled with one or more bounding boxes, wherein each of the one or more bounding boxes contains a sub-image. Furthermore, in addition to labeling one or more bounding boxes, the sample image may also be labeled with the type corresponding to each bounding box, where the type corresponding to each bounding box may refer to the type of the sub-image within that bounding box.

[0045] The method for obtaining the labeled bounding boxes and their corresponding types from the aforementioned sample images can be as follows: Perform 2D (two-dimensional) object detection on the original image to obtain sample images labeled with each bounding box and its corresponding type. 2D object detection can be implemented using a 2D object detection model. By using 2D object detection to obtain labeled data, each bounding box in the sample image is directly labeled, thus directly identifying the region of interest in the sample image. In subsequent processing, the images (or patches) within each bounding box can be directly obtained from the bounding boxes contained in the sample image. In subsequent processing, images within different bounding boxes can be used as positive and negative sample pairs for training a pre-defined model. Because the bounding boxes obtained by 2D object detection can directly filter out non-overlapping samples of interest, thereby reducing noise in the calculation of the contrastive loss function, the number of positive and negative sample pairs can be greatly reduced.

[0046] Alternatively, the annotation of each bounding box and the corresponding type of each bounding box in the above sample image can be obtained manually. This embodiment does not exhaustively list all annotation methods. The original image can be an image captured from a video, or an image captured by an acquisition device. The above is only an illustrative example. The original image can also be an image obtained by any other means. This embodiment does not exhaustively list all such methods.

[0047] Combination Figure 2 In general, the aforementioned preset model may include at least: an Encoder 201, a Decoder 202, an attention-weighted pooling layer 203, and a fully connected layer 204. Specifically, the encoder 201 contains multiple encoding sub-layers, such as... Figure 2 The diagram illustrates the first encoding sublayer 2011. The decoder 202 contains multiple decoding sublayers, such as... Figure 2 The first decoding sublayer 2021 is shown in the diagram.

[0048] Specifically, in the multiple coding sub-layers of the encoder described above, each coding sub-layer is used to extract features to obtain a feature map. The feature map extracted by the coding sub-layer is called the feature extraction map. The feature extraction map output by any coding sub-layer is reduced in size compared to the input image of that coding sub-layer. For example, the reduction ratio can be 1 / 2 or 1 / 4, etc. The reduction ratio can vary depending on the parameter settings of the encoder, which will not be exhaustively listed here.

[0049] In the above decoder, each of the multiple decoding sub-layers is used for feature filling to obtain a feature map. The feature map obtained by the decoding sub-layer is called the feature filling map. The feature filling map output by any decoding sub-layer is enlarged compared to the size of the input image of that decoding sub-layer. For example, the enlargement ratio can be 2 or 4, etc. The enlargement ratio can vary with different parameter settings, which will not be exhaustively listed here.

[0050] Each of the aforementioned encoding or decoding sub-layers may include Conv2d (Convolution2Dimension, two-dimensional convolution) and BatchNorm2d (Batch Normalization). Specifically, Conv2d in any encoding sub-layer can be used to perform convolution processing on the sample image to obtain a feature extraction map to be processed; BatchNorm2d in any encoding sub-layer can be used to perform batch normalization processing on the feature extraction map to be processed to obtain a feature extraction map of the sample image. Similarly, BatchNorm2d in any decoding sub-layer can be used to perform inverse batch normalization processing on the feature extraction map of the sample image to obtain a feature filling map to be processed; Conv2d in any decoding sub-layer can be used to perform deconvolution processing on the feature filling map to be processed of the sample image to obtain a feature filling map of the sample image.

[0051] The input layer of the aforementioned preset model can be the first encoding sub-layer of the encoder, and the output layer of the aforementioned preset model can be the last decoding sub-layer of the decoder. The first intermediate layer of the preset model can refer to any specific decoding sub-layer in the decoder of the preset model, excluding the last decoding sub-layer. The second intermediate layer of the aforementioned preset model can refer to the aforementioned fully connected layer.

[0052] Combination Figure 2 To illustrate this example, suppose... Figure 2 The decoding sublayer 2022 in the above-mentioned preset model is the first intermediate layer; the step of inputting a sample image into the preset model to obtain the feature map of the sample image output by the first intermediate layer of the preset model includes: inputting the sample image into the encoder 201 in the preset model to obtain the output of the encoder 201; and inputting the encoder output into the decoder 202 of the preset model to obtain the feature map of the sample image output by the decoding sublayer 2022 contained in the decoder 202. Figure 2 In this example, 210 represents the sample image.

[0053] Additionally, in this example, such as Figure 2 As shown, it may also include: a reconstructed image 211 of the sample image output by the output layer of the decoder of the preset model.

[0054] The size of the feature map of the sample image is smaller than that of the sample image. This can mean that the feature map of the sample image is reduced by a specified ratio compared to the sample image. In a preferred example, the specified ratio can be 1 / 8, that is, the final feature map of the sample image is 1 / 8 of the feature map of the sample image.

[0055] Furthermore, a neural network can be added between the specified decoding sub-layer in the aforementioned preset model and the previous decoding sub-layer. This neural network can specifically include a point-by-point self-attention global average pooling layer. The function of this point-by-point self-attention global average pooling layer is as follows: after performing convolution calculations on the input feature map, it calculates the weight (or self-attention) for each point to obtain a weighted feature map; then, it performs point-by-point multiplication between the weighted feature map and each corresponding point in the input feature map, followed by weight normalization to obtain a normalized feature map; finally, it compresses the normalized feature map to obtain a one-dimensional feature map, which is then used as the input feature map for the aforementioned specified decoding sub-layer. By adding the aforementioned point-by-point self-attention global average pooling layer, the final extracted feature map can contain more fine-grained and richer visual features.

[0056] Combination Figure 2 To illustrate further, the step of inputting the image within each of the multiple bounding boxes contained in the sample image into a preset model to obtain the feature vector corresponding to each bounding box output by the second intermediate layer of the preset model can refer to: inputting the image within the a-th bounding box in the sample image into the encoder 201 of the preset model to obtain the target feature map corresponding to the a-th bounding box output by the encoder 201; inputting the target feature map corresponding to the a-th bounding box into the attention weight pooling layer 203 of the preset model to obtain the intermediate result corresponding to the a-th bounding box output by the attention weight pooling layer 203; and inputting the intermediate result corresponding to the a-th bounding box into the fully connected layer 204 of the preset model (i.e., the aforementioned second intermediate layer) to obtain the feature vector Z corresponding to the a-th bounding box output by the fully connected layer 204 (i.e., the aforementioned second intermediate layer). Figure 2 In this example, 210 represents the image within the a-th bounding box. The a-th bounding box can be any one of all bounding boxes included in the sample image. Since the same processing is performed on each bounding box as on the a-th bounding box, it will not be elaborated upon individually. Ultimately, the feature vector corresponding to each bounding box needs to be obtained. Here, a is an integer greater than or equal to 1 and less than the number of bounding boxes. The number of bounding boxes can be two or more; this embodiment does not limit this. Combined with... Figure 3 For example, in Figure 3 In the sample image 301, there are two bounding boxes marked, as shown below. Figure 3 The bounding boxes 3011 and 3012 are shown. After processing by the preset model, the feature vector Z corresponding to each bounding box in the sample image output by the fully connected layer 204 of the preset model can be obtained. Figure 3The feature vector corresponding to bounding box 3011 is shown as Z01, and the feature vector corresponding to bounding box 3012 is shown as Z02.

[0057] In some possible implementations, generating the target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box may include: in the process of updating the consistency loss function for the i-th time, determining the i-th position in the feature map of the sample image based on the center point position of the i-th bounding box, where i is a positive integer; obtaining the feature vector to be compared at the i-th position from the feature map of the sample image; obtaining the vector difference corresponding to the i-th position based on the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position; obtaining the consistency loss function updated for the i-th time based on the vector difference corresponding to the i-th position and the consistency loss function updated for the (i-1)-th time; if the process of updating the consistency loss function for the i-th time is the last process of updating the consistency loss function, obtaining the target consistency loss function based on the consistency loss function updated for the i-th time, and obtaining the target loss function based on the target consistency loss function.

[0058] The i-th bounding box mentioned above is any one of the aforementioned bounding boxes. Since the processing for each bounding box is the same, it will not be described in detail.

[0059] The center point position of the i-th bounding box can refer to the position of the center point of the i-th bounding box in the sample image. In the process of updating the consistency loss function in the i-th iteration, determining the i-th position in the feature map of the sample image based on the center point position of the i-th bounding box can be as follows: In the process of updating the consistency loss function in the i-th iteration, arbitrarily select one unselected bounding box from all bounding boxes in the sample image as the i-th bounding box; in the feature map of the sample image, determine the target position corresponding to the position of the center point of the i-th bounding box in the sample image, and use this target position as the i-th position in the feature map of the sample image.

[0060] The step of obtaining the comparison feature vector at the i-th position from the feature map of the sample image can be: extracting the vector at the i-th position in the feature map of the sample image as the comparison feature vector at the i-th position.

[0061] The step of obtaining the vector difference at the i-th position based on the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position can include: calculating the dot product of the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position to obtain the dot product value at the i-th position; subtracting the dot product value at the i-th position from the first value to obtain the vector difference at the i-th position, that is, the vector difference at the i-th position is the dot product at the i-th position. The first value can be set according to the actual situation, for example, it can be 1, or it can be other values, which are not exhaustively listed in this embodiment.

[0062] In one possible example, the function to calculate the dot product can be `torch.dot()`. Let `Zi` represent the feature vector corresponding to the i-th bounding box, and `vectori - i` represent the feature vector to be compared at the i-th position. The dot product value at the i-th position can then be expressed as `torch.dot(vectori - i, Zi)`. Subtracting the dot product value at the i-th position from the first value to obtain the vector difference at the i-th position can be expressed as "1 - torch.dot(vectori - i, Zi)".

[0063] The step of obtaining the consistency loss function for the i-th update based on the vector difference corresponding to the i-th position and the consistency loss function of the (i-1)-th update can refer to adding the vector difference corresponding to the i-th position to the consistency loss function of the (i-1)-th update. For example, if the consistency loss function of the (i-1)-th update is represented as scale_consistent_loss(i-1) and the consistency loss function of the i-th update is represented as scale_consistent_loss_(i), then scale_consistent_loss(i) = scale_consistent_loss(i-1) + (1 - torch.dot(vector-i, Zi)).

[0064] Specifically, when i equals 1, the consistency loss function for the (i-1)th update can be empty or 0. When i is greater than 1, the consistency loss function for the (i-1)th update is obtained from the previous update consistency loss function. The generation method of the consistency loss function for the (i-1)th update is the same as that for the consistency loss function for the i-th update, so it will not be described in detail.

[0065] The determination method for whether the i-th update of the consistency loss function is the last update of the consistency loss function can be as follows: Determine whether there are any unprocessed bounding boxes within one or more bounding boxes of the sample image. If so, the i-th update of the consistency loss function is not the last update of the consistency loss function; otherwise, it is the last update of the consistency loss function. Alternatively, if the i-th update of the consistency loss function is not the last update of the consistency loss function, the (i+1)-th update of the consistency loss function can continue. In the (i+1)-th update of the consistency loss function, the (i+1)-th bounding box is selected from one or more bounding boxes in the sample image, and subsequent processing continues. This process is repeated until the last update of the consistency loss function is completed, ultimately yielding the target consistency loss function.

[0066] The phrase "obtaining the target consistency loss function based on the consistency loss function of the i-th update" can mean directly using the consistency loss function of the i-th update as the target consistency loss function. Alternatively, it can mean dividing the consistency loss function of the i-th update by the number of bounding boxes to obtain the target consistency loss function.

[0067] In some possible examples, the above-described method of obtaining the target loss function based on the target consistency loss function can be described as directly using the target consistency loss function as the target loss function.

[0068] As can be seen, by adopting the above scheme, the feature maps of the sample images output by the first and second intermediate layers of the preset model, as well as the feature vectors corresponding to each bounding box, can be combined to obtain the target consistency loss function. In this way, relevant information about the consistency between the feature vectors of the bounding boxes and the feature vectors at corresponding positions in the feature maps can be added to the target loss function, ensuring that the feature maps processed by the target model are more accurate after updating the preset model based on the target loss function.

[0069] In some possible implementations, the method further includes: generating a first target contrast loss function based on the feature vectors corresponding to different bounding boxes;

[0070] The step of obtaining the target loss function based on the target consistency loss function includes: obtaining the target loss function based on the target consistency loss function and the first target comparison loss function.

[0071] Specifically, the different bounding boxes mentioned above can form positive and negative sample pairs, and the first target contrast loss function mentioned above can refer to the contrast loss function between positive and negative sample pairs.

[0072] The step of generating a first target contrastive loss function based on the feature vectors corresponding to different bounding boxes may include: in the current calculation of the contrastive loss sub-function, selecting the x-th bounding box that is different from the target bounding box, where x is an integer; calculating the current loss sub-function between the feature vector corresponding to the x-th bounding box and the feature vector corresponding to the target bounding box; adding the current loss sub-function to the previous contrastive loss sub-function to obtain the current contrastive loss sub-function; and if the current calculation of the contrastive loss sub-function is the last calculation of the contrastive loss function, using the current contrastive loss sub-function as the first target contrastive loss function.

[0073] Here, the target bounding box can be a pre-specified bounding box among one or more bounding boxes in the sample image. For example, it can be any one of the one or more bounding boxes in the sample image that is pre-specified as the target bounding box. Selecting the x-th bounding box that is different from the target bounding box can mean selecting any one of the other bounding boxes that has not been compared with the target bounding box from all bounding boxes as the x-th bounding box, and forming a positive and negative sample pair with the x-th bounding box and the target bounding box. The number of other bounding boxes can be one or more, and these other bounding boxes refer to bounding boxes other than the target bounding box.

[0074] The step of calculating the current loss sub-function between the feature vector corresponding to the x-th bounding box and the feature vector corresponding to the target bounding box may include: calculating the contrast loss between the feature vector corresponding to the x-th bounding box and the feature vector corresponding to the target bounding box, and using this contrast loss as the current loss sub-function. The method for calculating the contrast loss can be set according to the actual situation. For example, any one of the following methods can be used to calculate the contrast loss: pairwise ranking loss, triplet loss, etc. This embodiment does not exhaustively list all possible methods.

[0075] Specifically, if the current iteration is the first iteration, the previous comparison loss function can be empty or 0. If the current iteration is not the first iteration, the generation method of the previous comparison loss function is the same as that of the current comparison loss function, so it will not be elaborated further.

[0076] The method for determining whether the current calculation of the contrast loss sub-function is the same as the last calculation of the contrast loss function can be as follows: Determine whether there are other bounding boxes in one or more bounding boxes of the sample image that have not been compared with the target bounding box. If so, the current calculation of the contrast loss sub-function is not the same as the last calculation of the contrast loss function; otherwise, it is. Alternatively, if the current calculation of the contrast loss sub-function is not the same as the last calculation of the contrast loss function, the next calculation of the contrast loss sub-function can continue, and so on, until all bounding boxes have been processed, ultimately yielding the first target contrast loss function.

[0077] Still combined Figure 3 For example, assuming that the feature vector corresponding to the bounding box 3011 of the sample image 301 is Z01 and the feature vector corresponding to the bounding box 3012 is Z02, the bounding box 3011 can be used as the target bounding box and the bounding box 3012 can be used as the x-th bounding box. Since there are only two bounding boxes, only one processing is required. Through one processing, the contrast loss between Z01 and Z02 can be obtained to obtain the current loss sub-function. The current contrast loss sub-function is used as the first target contrast loss function.

[0078] The step of obtaining the target loss function based on the target consistency loss function and the first target comparison loss function can be achieved by weighted summation of the target consistency loss function and the first target comparison loss function. In the weighted summation process, the weights corresponding to the target consistency loss function and the first target comparison loss function can be preset according to actual conditions, and the weights corresponding to the target consistency loss function and the first target comparison loss function can be the same or different.

[0079] As can be seen, by adopting the above scheme, a first target contrast loss function is further added, derived from the feature vectors of different bounding boxes in the sample image, based on the target consistency loss function. This ensures that the final target loss function not only represents the target consistency loss but also incorporates the contrast states between different bounding boxes, guaranteeing the accuracy of the target loss function and resulting in accurate updates to the final target model.

[0080] In some possible implementations, the method further includes: inputting the image within each first bounding box of one or more first bounding boxes of the first enhanced image and the image within each second bounding box of one or more second bounding boxes of the second enhanced image into the preset model, respectively, to obtain a first feature vector corresponding to each first bounding box and a second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model, wherein the first enhanced image and the second enhanced image are obtained by processing the sample image using different enhancement methods; and obtaining a cyclic loss function based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box.

[0081] The step of obtaining the target loss function based on the target consistency loss function and the first target comparison loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, and the cyclic loss function.

[0082] The acquisition methods for the first enhanced image and the second enhanced image may include: processing the sample image using a first enhancement method to obtain the first enhanced image; and processing the sample image using the first enhancement method to obtain the second enhanced image. The first enhancement method is different from the second enhancement method.

[0083] The first enhancement method can be any two different enhancement methods selected from a variety of optional enhancement methods. These optional enhancement methods may include: JPEG (Joint Photographic Experts Group) random quality factor compression; random noise addition (random Gaussian noise, Poisson noise, or patch noise); random mosaic; and random sharpening and / or blurring. It should be understood that in addition to the methods exemplified above, other image enhancement methods may also be available, which will not be elaborated upon here.

[0084] It should be noted that, as described in the foregoing embodiments, the sample image includes one or more labeled bounding boxes, and the type of each bounding box; similarly, the first enhanced image and the second enhanced image also have corresponding annotations. For example, the first enhanced image also includes one or more first bounding boxes, and the type of each first bounding box is labeled. The number of first bounding boxes in the first enhanced image is the same as the number of bounding boxes in the sample image, and each first bounding box in the first enhanced image has a bounding box located at the same position in the sample object. The annotations in the second enhanced image are similar to those in the first enhanced image and will not be repeated. Figure 3 Specifically, the first enhanced image 302 is obtained based on the sample image 301, and the first enhanced image 302 includes two first bounding boxes, as shown below. Figure 3 The first bounding box 3021 and the second bounding box 3022 are shown. Similarly, the second enhanced image 303 also includes two second bounding boxes, as shown below. Figure 3 The second bounding box 3031 and the second bounding box 3032 are shown. The enhancement methods for the first enhanced image 302 and the second enhanced image 303 are not limited, as long as they differ, they are within the protection scope of this example.

[0085] The above-mentioned inputting the images within each first bounding box of one or more first bounding boxes of the first enhanced image and the images within each second bounding box of one or more second bounding boxes of the second enhanced image into the preset model to obtain the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model can be: inputting the images within each first bounding box of one or more first bounding boxes of the first enhanced image into the preset model to obtain the first feature vector corresponding to each first bounding box output by the second intermediate layer of the preset model; and inputting the images within each second bounding box of one or more second bounding boxes of the second enhanced image into the preset model to obtain the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model. Still in conjunction with... Figure 3 To illustrate, in Figure 3 After inputting the image of the first bounding box 3021 of the first enhanced image 302 into the preset model, the first feature vector corresponding to the first bounding box 3021 is obtained (i.e., Figure 3 As shown in Z11); using the same method, the first feature vector corresponding to the first bounding box 3022 of the first enhanced image 302 can be obtained (i.e., Figure 3 As shown in Z12), and obtain the second feature vector corresponding to the second bounding box 3031 of the second enhanced image 303 (i.e., Figure 3 The second feature vector (i.e., the one shown is Z21), and the second feature vector corresponding to the second bounding box 3032. Figure 3 (Z22 as shown).

[0086] The target loss function is obtained based on the target consistency loss function, the first target comparison loss function, and the cyclic loss function. This can be achieved by weighted summation of the target consistency loss function, the first target comparison loss function, and the cyclic loss function. In the weighted summation process, the weights corresponding to the target consistency loss function, the first target comparison loss function, and the cyclic loss function can be preset according to the actual situation and can be the same or different.

[0087] In other possible examples, the method may also include: inputting a first enhanced image into the preset model to obtain a feature map of the first enhanced image output by the first intermediate layer of the preset model; and inputting a second enhanced image into the preset model to obtain a feature map of the second enhanced image output by the first intermediate layer of the preset model. The processing of obtaining the feature map of the first enhanced image from the first intermediate layer of the preset model, and the processing of obtaining the feature map of the second enhanced image from the first intermediate layer of the preset model, are similar to the aforementioned processing of obtaining the feature map of the sample image from the first intermediate layer of the preset model, and will not be elaborated upon here.

[0088] Unlike the previous embodiments, in this example, after obtaining the feature maps of the first enhanced image and the second enhanced image, a consistency loss function corresponding to the first enhanced image can be generated based on the feature maps of the first enhanced image and the feature vectors corresponding to each first bounding box. Similarly, a consistency loss function corresponding to the second enhanced image can be generated based on the feature maps of the second enhanced image and the feature vectors corresponding to each second bounding box. The process of generating the consistency loss function of the first enhanced image based on the feature maps of the first enhanced image and the feature vectors corresponding to each first bounding box is similar to the process of obtaining the target consistency loss function in the previous embodiments and will not be repeated. Likewise, the process of generating the consistency loss function of the second enhanced image based on the feature maps of the second enhanced image and the feature vectors corresponding to each second bounding box is also similar to the process of obtaining the target consistency loss function in the previous embodiments and will not be repeated.

[0089] Furthermore, the process of obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function may further include: performing a weighted summation based on the target consistency loss function, the consistency loss function corresponding to the first enhanced image, the consistency loss function corresponding to the second enhanced image, and the cyclic loss function to obtain the target loss function. In the weighted summation process, the weights corresponding to the target consistency loss function, the consistency loss function corresponding to the first enhanced image, the consistency loss function corresponding to the second enhanced image, and the cyclic loss function can be preset according to actual conditions and can be the same or different.

[0090] In some possible examples, after obtaining the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model, the process may further include: generating a first positive-negative contrast loss function based on the feature vectors corresponding to different first bounding boxes, and generating a second positive-negative contrast loss function based on the feature vectors corresponding to different second bounding boxes. The specific processing for generating the first positive-negative contrast loss function and the specific processing for generating the second positive-negative contrast loss function are similar to the specific processing for generating the first target contrast loss function in the aforementioned embodiments, and therefore will not be elaborated upon.

[0091] Furthermore, obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function can include: performing a weighted summation of the target consistency loss function, the consistency loss function corresponding to the first enhanced image, the consistency loss function corresponding to the second enhanced image, the first target contrast loss function, the first positive-negative contrast loss function, the second positive-negative contrast loss function, and the cyclic loss function to obtain the target loss function. In the weighted summation process, the weights corresponding to each function can be preset according to the actual situation and can be the same or different.

[0092] As can be seen, by adopting the above scheme, a cyclic loss function calculated from the first bounding box and the second bounding box corresponding to the two enhanced images can be introduced on the basis of the target consistency loss function and the first target contrast loss function. This makes the composition dimension of the target loss function richer, improves the accuracy of the target loss function, and further improves the accuracy of the target model.

[0093] In some possible implementations, obtaining the cyclic loss function based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box includes:

[0094] In the m-th calculation of the cumulative sum of similarity differences, the k-th first bounding box and the j-th first bounding box are selected, and the k-th second bounding box and the j-th second bounding box are also selected; wherein, the position of the k-th first bounding box in the first enhanced image is the same as the position of the k-th second bounding box in the second enhanced image, and the position of the j-th first bounding box in the first enhanced image is the same as the position of the j-th second bounding box in the second enhanced image, k and j are different, and m, k and j are all positive integers;

[0095] A first similarity is obtained based on the first feature vector corresponding to the kth first bounding box and the second feature vector corresponding to the jth second bounding box, and a second similarity is obtained based on the first feature vector corresponding to the jth first bounding box and the second feature vector corresponding to the kth second bounding box.

[0096] Calculate the m-th similarity difference between the first similarity and the second similarity;

[0097] Based on the sum of the m-th similarity difference and the (m-1)-th similarity difference, the sum of the m-th similarity differences is obtained;

[0098] If the process of calculating the cumulative sum of similarity differences in the mth time is the last process of calculating the cumulative sum of similarity differences, the cyclic loss function is obtained based on the cumulative sum of similarity differences in the mth time.

[0099] In some possible examples, the aforementioned k-th first bounding box and k-th second bounding box can be pre-specified; that is, the aforementioned k-th first bounding box is a specified first bounding box, and the aforementioned k-th second bounding box is a specified second bounding box. It should be noted that the position of the specified first bounding box in the first enhanced image and the position of the specified second bounding box in the second enhanced image are the same.

[0100] Accordingly, selecting the k-th first bounding box, the j-th first bounding box, and the k-th second bounding box, and the j-th second bounding box can be: selecting, from one or more first bounding boxes in the first enhanced image, other first bounding boxes that are different from the specified first bounding box and have not been selected before, as the j-th first bounding box; and selecting, from one or more second bounding boxes in the second enhanced image, a second bounding box in the same position as the j-th first bounding box, as the j-th second bounding box. For consistency, the specified first bounding box will still be referred to as the k-th first bounding box, and the specified second bounding box will still be referred to as the k-th second bounding box.

[0101] In some other possible examples, selecting the kth first bounding box, the jth first bounding box, and the kth second bounding box, the jth second bounding box may include: determining two first bounding boxes that have not been selected simultaneously from the one or more first bounding boxes of the first enhanced image as the kth first bounding box and the jth first bounding box, respectively; determining a second bounding box with the same position as the jth first bounding box from the one or more second bounding boxes of the second enhanced image as the jth second bounding box; and determining a second bounding box with the same position as the kth first bounding box as the kth second bounding box.

[0102] The step of obtaining the first similarity based on the first feature vector corresponding to the k-th first bounding box and the second feature vector corresponding to the j-th second bounding box can be achieved by: calculating the dot product of the first feature vector corresponding to the k-th first bounding box and the second feature vector corresponding to the j-th second bounding box to obtain the first dot product value, and using this first dot product value as the first similarity. For example, the function for calculating the dot product is the same as in the previous embodiment, and the torch.dot() function can be used; the first feature vector corresponding to the k-th first bounding box is represented as P1_vector_Zk, and the second feature vector corresponding to the j-th second bounding box is represented as P2_vector_Zj. The above method for obtaining the first dot product value can be expressed as torch.dot(P1_vector_Zk, P2_vector_Zj).

[0103] The step of obtaining the second similarity based on the first feature vector corresponding to the j-th first bounding box and the second feature vector corresponding to the k-th second bounding box can be achieved by: calculating the dot product of the first feature vector corresponding to the j-th first bounding box and the second feature vector corresponding to the k-th second bounding box, obtaining the second dot product value, and using this second dot product value as the second similarity. For example, the first feature vector corresponding to the j-th first bounding box can be represented as P1_vector_Zj, and the second feature vector corresponding to the k-th second bounding box can be represented as P2_vector_Zk. The calculation method for obtaining the second dot product value can be expressed as torch.dot(P1_vector_Zj, P2_vector_Zk).

[0104] Calculating the m-th similarity difference between the first similarity and the second similarity can be achieved by subtracting the first similarity from the second similarity to obtain the original difference, and then using the absolute value of the original difference as the m-th similarity difference. For example, if the first similarity is denoted as sim_P1k_P2j and the second similarity as sim_P1j_P2k, then the m-th similarity difference can be equal to abs(sim_P1k_P2j - sim_P1j_P2k), where the abs() function represents the function for calculating the absolute value.

[0105] The method of obtaining the cumulative sum of similarity differences based on the m-th similarity difference and the cumulative sum of similarity differences of the (m-1)-th similarity difference can be achieved by adding the m-th similarity difference to the cumulative sum of similarity differences of the (m-1)-th similarity difference.

[0106] Specifically, when m equals 1, the cumulative sum of the (m-1)th similarity difference can be 0. When m is greater than 1, the method for generating the cumulative sum of the (m-1)th similarity difference is the same as the method for generating the cumulative sum of the mth similarity difference, and therefore will not be elaborated upon.

[0107] The determination method for whether the m-th calculation of the cumulative sum of similarity differences is the last calculation of the cumulative sum of similarity differences can be as follows: Determine whether there are any unextracted first bounding boxes within one or more first bounding boxes of the first enhanced image. If so, the m-th calculation of the cumulative sum of similarity differences is not the last calculation of the cumulative sum of similarity differences; if not, the m-th calculation of the cumulative sum of similarity differences is the last calculation of the cumulative sum of similarity differences. Alternatively, if the m-th calculation of the cumulative sum of similarity differences is not the last calculation of the cumulative sum of similarity differences, the (m+1)-th calculation of the cumulative sum of similarity differences can continue, and so on, until the last calculation of the cumulative sum of similarity differences is completed. The last cumulative sum of similarity differences is then used as the cyclic loss function.

[0108] Combination Figure 3 To illustrate, due to Figure 3Each enhanced image contains only two bounding boxes, so only one loop is needed. In this loop, a first similarity is obtained based on the first feature vector Z11 corresponding to the first bounding box 3021 and the second feature vector Z22 corresponding to the second bounding box 3032. A second similarity is obtained based on the first feature vector Z12 corresponding to the first bounding box 3022 and the second feature vector Z21 corresponding to the second bounding box 3031. The similarity difference between the first similarity and the second similarity is calculated, and this similarity difference is directly used as the cumulative sum of similarity differences. Then, the cumulative sum of similarity differences is used as the loop loss function.

[0109] The step of using the sum of the last similarity differences as the cyclic loss function can include: dividing the sum of the last similarity differences by the number of iterations to obtain the cyclic loss function, or directly using the sum of the last similarity differences as the cyclic loss function. The number of iterations can be determined based on the actual situation and is related to the number of bounding boxes; this embodiment does not limit it.

[0110] As can be seen, by adopting the above scheme, in the processing of the cyclic loss function calculated for the first and second bounding boxes corresponding to the two enhanced images respectively, two first and second bounding boxes at the same position are selected and their similarity is calculated. The similarity difference is obtained based on the similarity between them, and all similarity differences are accumulated to finally obtain the cyclic loss function. This makes the composition dimension of the target loss function richer, and the similarity comparison is also performed between different images, which improves the accuracy of the target loss function and further improves the accuracy of the target model.

[0111] In some possible implementations, the method further includes: obtaining a second target contrast loss function based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box;

[0112] The step of obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, and the cyclic loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, and the second target comparison loss function.

[0113] The second objective contrast loss function can refer to the contrast loss function between positive sample pairs.

[0114] In some possible examples, the step of obtaining the second target contrastive loss function based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box includes:

[0115] In the nth generation of the positive sample contrast loss function, the nth bounding box, the nth first bounding box, and the nth second bounding box are selected; where n is a positive integer, and the positions of the nth bounding box in the sample image, the nth first bounding box in the first enhanced image, and the nth second bounding box in the second enhanced image are the same;

[0116] A first positive sample contrast loss is calculated based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box; a second positive sample contrast loss is calculated based on the feature vector corresponding to the nth bounding box and the second feature vector corresponding to the nth second bounding box; and a third positive sample contrast loss is calculated based on the feature vector corresponding to the nth first bounding box and the second feature vector corresponding to the nth second bounding box.

[0117] Based on the first positive sample contrast loss, the second positive sample contrast loss, and the third positive sample contrast loss, the current positive sample contrast loss sub-function is obtained;

[0118] The nth positive sample contrast loss function is obtained by adding the current positive sample contrast loss function to the (n-1)th positive sample contrast loss function.

[0119] If the nth generation positive sample contrast loss function is the same as the last generation positive sample contrast loss function, the second target contrast loss function is generated based on the nth positive sample contrast loss function.

[0120] The selection of the nth bounding box, the nth first bounding box, and the nth second bounding box can refer to selecting any unaligned bounding box from all bounding boxes in the sample image as the nth bounding box, selecting the nth first bounding box in the first enhanced image that is at the same position as the nth bounding box, and selecting the nth second bounding box in the second enhanced image that is at the same position as the nth bounding box. The nth bounding box, the nth first bounding box, and the nth second bounding box selected in this way form positive sample pairs in pairs.

[0121] The calculation of the first positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box may include: calculating the positive sample contrast loss between the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box, and using this contrast loss as the first positive sample contrast loss. The specific processing methods for calculating the second positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the second feature vector corresponding to the nth second bounding box, and for calculating the third positive sample contrast loss based on the feature vector corresponding to the nth first bounding box and the second feature vector corresponding to the nth second bounding box, are similar to those for calculating the first positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box, and therefore will not be elaborated upon. The method for calculating the positive sample contrast loss can be set according to the actual situation; for example, the same method as in the aforementioned embodiments can be used to calculate the contrast loss, which will not be elaborated upon here.

[0122] The phrase "obtaining the current positive sample comparison loss sub-function based on the first positive sample comparison loss, the second positive sample comparison loss, and the third positive sample comparison loss" can refer to: weighted summation of the first positive sample comparison loss, the second positive sample comparison loss, and the third positive sample comparison loss to obtain the current positive sample comparison loss sub-function.

[0123] The determination method for whether the nth generation positive sample contrast loss function is the last generation positive sample contrast loss function can be as follows: Determine whether there are any unselected bounding boxes within one or more bounding boxes of the sample image. If so, the nth generation positive sample contrast loss function is not the last generation positive sample contrast loss function; if not, it is. Alternatively, if the nth generation positive sample contrast loss function is not the last generation positive sample contrast loss function, the processing of the (n+1)th generation positive sample contrast loss function can continue, and so on, until the processing of the last generation positive sample contrast loss function is completed. The last generation positive sample contrast loss function is then used as the second target contrast loss function. Still combined with... Figure 3 To illustrate, due to Figure 3Each image contains only two bounding boxes, so only two iterations are needed. Taking the first iteration as an example, the bounding box 3011 of sample image 301, the first bounding box 3021 of the first enhanced image 302, and the second bounding box 3031 of the second enhanced image 303 are selected. Then, based on the feature vector Z01 corresponding to the bounding box 3011 and the first feature vector Z11 corresponding to the first bounding box 3021, the first positive sample contrast loss is calculated. Based on the feature vector Z01 corresponding to the bounding box 3011 and the second feature vector Z21 corresponding to the second bounding box 3031, the second positive sample contrast loss is calculated. Based on the feature vector Z01 corresponding to the bounding box 3011 and the second feature vector Z21 corresponding to the second bounding box 3031, the second positive sample contrast loss is calculated. The third positive sample contrast loss is calculated using the first feature vector Z11 and the second feature vector Z21 corresponding to the second bounding box 3031. Based on the first positive sample contrast loss, the second positive sample contrast loss, and the third positive sample contrast loss, the current positive sample contrast loss sub-function is obtained, and the current positive sample contrast loss sub-function is directly used as the nth positive sample contrast loss function. Then, the second loop is executed to select the bounding box 3012 of the sample image 301, the first bounding box 3022 of the first enhanced image 302, and the second bounding box 3032 of the second enhanced image 303. This will not be repeated later. Finally, the second target contrast loss function can be obtained.

[0124] The step of obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, and the second target contrast loss function can refer to: weighting and summing the target consistency loss function, the first target contrast loss function, the cyclic loss function, and the second target contrast loss function respectively to obtain the target loss function. It should be understood that the foregoing embodiments also mentioned the possibility of generating a consistency loss function corresponding to the first enhanced image, a consistency loss function corresponding to the second enhanced image, a first positive-negative contrast loss function, and a second positive-negative contrast loss function; correspondingly, all of the above types of loss functions can be weighted and calculated separately and then summed to obtain the final target loss function.

[0125] As can be seen, by adopting the above scheme, the feature vectors of bounding boxes at the same position in the sample image, the first enhanced image, and the second enhanced image can be compared to obtain the contrast loss for each positive sample. Then, based on the contrast losses of each positive sample, a second target contrast loss function is obtained, thereby increasing the contrast between positive sample pairs and ensuring the accuracy of the final contrast loss function. Furthermore, in the calculation of the target loss function, by adding the second target contrast loss function constructed from the positive sample contrast losses, the dimensions covered by the target loss function can be made more comprehensive, ensuring the accuracy of the target loss function and thus ensuring the accuracy of the target model.

[0126] In some possible implementations, after inputting the first enhanced image and the second enhanced image into the preset model respectively, the method further includes: obtaining the reconstructed images of the first enhanced image and the second enhanced image respectively output by the output layer of the preset model; obtaining a first reconstruction loss function based on the sample image and the reconstructed image of the first enhanced image; obtaining a second reconstruction loss function based on the sample image and the reconstructed image of the second enhanced image; and obtaining a reconstruction loss function based on the first reconstruction loss function and the second reconstruction loss function.

[0127] The step of obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, and the second target comparison loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, the second target comparison loss function, and the reconstruction loss function.

[0128] The description of the output layer of the preset model is the same as in the previous embodiment; it can be the last decoding sub-layer of the decoder in the preset model. The specific processing methods for obtaining the reconstructed images of the first enhanced image and the second enhanced image output by the output layer of the preset model are also the same as the specific processing methods for the reconstructed images of the sample images output by the output layer of the preset model in the previous embodiment, and therefore will not be repeated.

[0129] The step of obtaining the first reconstruction loss function based on the sample image and the reconstructed image of the first enhanced image may include: obtaining the first reconstruction loss function based on the similarity between the sample image and the reconstructed image of the first enhanced image. The specific calculation function of the aforementioned first reconstruction loss function is set according to the actual situation, and for example, any one of the following can be used: SSIM loss function (Structural Similarity Index), LPIPS loss function (Learned Perceptual Image PatchSimilarity), etc., which are not exhaustively listed in this embodiment.

[0130] The processing method for obtaining the second reconstruction loss function based on the sample image and the second enhanced image is similar to the processing method for obtaining the first reconstruction loss function based on the sample image and the first enhanced image, and therefore will not be repeated. The purpose of constructing the reconstruction loss function is to enable the trained target model to reconstruct the original, defect-free, and complete image from a corrupted, noisy, and incomplete image through an encoder-decoder process.

[0131] The target loss function is obtained based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, the second target contrast loss function, and the reconstruction loss function. This can be achieved by weighting and summing the target consistency loss function, the first target contrast loss function, the cyclic loss function, the second target contrast loss function, and the reconstruction loss function. It should be understood that the foregoing embodiments also mentioned the possibility of generating consistency loss functions corresponding to the first enhanced image, consistency loss functions corresponding to the second enhanced image, the first positive-negative contrast loss function, and the second positive-negative contrast loss function; correspondingly, all of the above types of loss functions can be weighted and calculated separately, and then summed to obtain the final target loss function.

[0132] It should also be noted that other types of loss functions can be added to the solution provided in this embodiment. For example, an object detection loss function can also be included. The object detection loss function may include a bounding box regression loss function, a confidence loss function, etc. This embodiment does not limit the construction method of the bounding box regression loss function and the confidence loss function. After obtaining the object detection loss function, the object detection loss function and all types of loss functions obtained in the above embodiments can be further weighted and summed to obtain the target loss function.

[0133] As can be seen, by adopting the above scheme, in the process of calculating the target loss function, by adding the reconstruction loss function constructed from the reconstructed image obtained from the final output layer of the preset model and the sample image, the dimensions covered by the target loss function can be made more comprehensive, thus ensuring the accuracy of the target loss function and consequently ensuring the accuracy of the target model.

[0134] In some possible implementations, updating the preset model based on the target loss function may refer to updating the parameters of the preset model based on the target loss function. The parameters of the preset model may include the parameters of each layer of the aforementioned prediction model.

[0135] The aforementioned convergence conditions can be set according to the actual situation. For example, they may include at least one of the following: the result of the target loss function no longer changes, the preset number of model updates reaches the preset number, the result of the target loss function is less than a specified value, etc. The convergence conditions are not exhaustively listed here.

[0136] In some possible implementations, the method may further include: inputting a target image into a target model to obtain a feature map of the target image output by the target model, wherein the size of the feature map of the target image is smaller than that of the target image.

[0137] The step of inputting the target image into the target model to obtain the feature map of the target image output by the target model can specifically refer to: inputting the target image into the target model to obtain the feature map of the target image output by the first intermediate layer of the target model. Simultaneously with inputting the target image into the target model to obtain the feature map of the target image output by the target model, the step may also include: obtaining a reconstructed image of the target image output by the output layer of the target model, wherein the size of the reconstructed image of the target image is the same as that of the target image.

[0138] The feature map of the target image mentioned above can be a feature map of a specified ratio, which can be 1 / 8, that is, the feature map of the target image is 1 / 8 of the feature map of the target image.

[0139] After obtaining the reconstructed image of the target image output by the target model, the process may further include: detecting the reconstructed image of the target image to obtain one or more target bounding boxes contained in the target image, and the type corresponding to each of the one or more target bounding boxes. Correspondingly, after obtaining the feature map of the target image output by the target model, the process may further include: determining the center point coordinates of each target bounding box; extracting the target feature vector corresponding to each target bounding box from the feature map of the target image based on the center point coordinates of each target bounding box; performing person re-identification (REID) based on the target feature vector corresponding to each target bounding box; and obtaining the tracking matching result corresponding to each target bounding box. Each target bounding box may correspond to a target object, and different target bounding boxes may correspond to different target objects; furthermore, obtaining the tracking matching result corresponding to each target bounding box may refer to obtaining the tracking matching result of the target object corresponding to each target bounding box.

[0140] The target image mentioned above can be any frame in a series of video frames. It should be understood that if the image processing method of this embodiment is applied to consecutive video frames, each video frame can be used as the current target image for the above processing. Since the processing performed on each video frame is the same as that performed on the aforementioned target image, it will not be described in detail. Furthermore, after performing the same processing on consecutive video frames, the tracking and matching results for each target to be tracked can be obtained.

[0141] Because the training process incorporates various losses, including contrast loss for positive and negative samples, consistency loss between feature maps and feature vectors, contrast loss between positive samples, and reconstruction loss, the final target model can contain not only the abstract semantic information of deep networks but also fine-grained information from low-level networks, such as color, texture, and shape. Specifically, updating the preset model based on contrast loss for positive and negative samples, consistency loss between feature maps and feature vectors, and contrast loss between positive samples allows the model to extract features with coarser granularity, higher semantic level, and greater abstraction. Updating the preset model based on reconstruction loss allows the model to extract features with finer granularity and richer detail. Therefore, by using the scheme provided in this embodiment for model training, the target model can process the target image to obtain a feature map smaller than the target image. Using this feature map for subsequent multi-target tracking calculations can ensure accuracy while reducing computational load, thereby improving processing efficiency.

[0142] The beneficial effects of the model training method provided in this embodiment will be explained in detail below with reference to relevant technologies: Visual target tracking refers to the detection, extraction, recognition, and tracking of moving targets in an image sequence. Based on the number of targets being tracked, tracking algorithms can be divided into single-target tracking and multi-target tracking. Compared to single-target tracking, multi-target tracking is a more complex and difficult problem. Multi-target tracking requires consideration of data such as the position and size of multiple independent targets in the video sequence. Changes in the appearance of each target, different motion patterns, the influence of dynamic lighting, and situations such as occlusion, merging, and separation between targets are all challenges in multi-target tracking. In related technologies, multi-target tracking algorithms, based on image feature extraction (custom visual features such as color histogram statistics, color linear correlation coefficients, etc.) and optimization (probabilistic graph optimization, minimum energy optimization, etc.), typically suffer from slow processing speed and a tendency to mismatch with intersecting vehicles.

[0143] To address the aforementioned issues, related technologies have further proposed a multi-object tracking method based on deep learning and data-driven approaches. This method primarily involves: extracting object detection boxes, extracting object feature vectors, and matching the currently detected object with the current tracking trajectory to form a continuous trajectory. However, to achieve optimal tracking and matching results, the above methods require joint learning of object detection and deep feature extraction. Both object detection and deep feature extraction are purely supervised training processes. Multi-object tracking algorithms, on the other hand, require labeled data based on video sequences. This includes not only labeled detection information (bounding boxes, categories) but also labeled object identity information for target instance discrimination training (the purpose being to distinguish different targets of the same category, such as differentiating two different vehicles belonging to the same category in the current image frame). It can be seen that the multi-object tracking method based on deep learning and data-driven approaches provided by the aforementioned technologies is a supervised training method, requiring manually labeled multiple video sequences as training data. This results in time-consuming and labor-intensive labeling work, and also demands high labeling quality.

[0144] The model training method provided in this embodiment can extract feature vectors from each bounding box of the sample image through a preset model, and generate feature maps from the sample image through the preset model. Then, loss calculation is performed based on the feature vectors and feature maps. In this way, self-supervised training can be completed. Training can be completed using only images. Only bounding boxes and type annotations are needed for the images, without the need for annotation of target objects. This reduces the reliance on manual data annotation, lowers the requirements for data annotation, and improves processing efficiency.

[0145] The second aspect of this disclosure provides an image processing method, such as... Figure 4 As shown, it includes:

[0146] S401. Input the target image into the target model to obtain the feature map of the target image output by the target model, wherein the size of the feature map of the target image is smaller than the target image, the feature map of the target image is used for tracking and matching, and the target model is obtained based on the model training method.

[0147] The information processing method provided in this embodiment can be executed by an electronic device, which can be a terminal device, such as a tablet computer, a laptop computer, a mobile phone, a desktop computer, or any other type.

[0148] The target image mentioned above can be any frame in a series of video frames. It should be understood that if the image processing method of this embodiment is applied to a series of video frames, each video frame can be treated as a target image and the above processing can be performed. Since the processing performed on each video frame is the same as that performed on the target image, it will not be described in detail.

[0149] As can be seen, by adopting the above scheme, the target image can be processed by the target model to obtain the feature map of the target image, and then the tracking and matching process can be performed based on the feature map. Since the size of the feature map is smaller than the original target image, the subsequent tracking and matching process can reduce the amount of computation and improve the processing efficiency.

[0150] In some possible implementations, the target image is input into the target model to obtain the feature map of the target image output by the target model. Specifically, this may refer to: inputting the target image into the target model to obtain the feature map of the target image output by the first intermediate layer of the target model.

[0151] The composition of the target model can be the same as the preset model in the first aspect embodiment described above, such as including an Encoder, a Decoder, an attention weight pooling layer, and a fully connected layer, which will not be repeated here. The first intermediate layer of the target model can specifically refer to a designated decoding sublayer of the decoder of the target model. The definition of this designated decoding sublayer is the same as in the first aspect embodiment described above, and will not be repeated here.

[0152] The feature map of the target image can be a feature map of a specified ratio, which is the same as the definition in the first aspect embodiment mentioned above. For example, it can be 1 / 8, that is, the feature map of the target image is 1 / 8 of the feature map of the target image.

[0153] In some possible implementations, after inputting the target image into the target model, the method further includes: obtaining a reconstructed image of the target image output by the target model; and detecting one or more target bounding boxes contained in the reconstructed image of the target image.

[0154] After obtaining the feature map of the target image output by the target model, the method further includes: determining the center point coordinates of each target bounding box in the one or more target bounding boxes; extracting the target feature vector corresponding to each target bounding box from the feature map of the target image based on the center point coordinates of each target bounding box; and obtaining the tracking matching result corresponding to each target bounding box based on the target feature vector corresponding to each target bounding box.

[0155] In addition, while obtaining one or more target bounding boxes contained in the target image, the method may also include obtaining the type corresponding to each target bounding box in the one or more target bounding boxes.

[0156] Wherein, obtaining the reconstructed image of the target image output by the target model can refer to obtaining the reconstructed image of the target image output by the output layer of the target model.

[0157] Determining the center point coordinates of each target bounding box can refer to determining the center point coordinates of each target bounding box in the reconstructed image of the target image.

[0158] The step of extracting the target feature vector corresponding to each target bounding box from the feature map of the target image based on the center point coordinates of each target bounding box may include: determining the b-th position point in the feature map of the target image corresponding to the center point coordinates of the b-th target bounding box based on the center point coordinates of the b-th target bounding box in the reconstructed image of the target image, and extracting the feature vector of the b-th position point in the feature map of the target image as the target feature vector corresponding to the b-th target bounding box.

[0159] The step of obtaining the tracking and matching result corresponding to each target bounding box based on the target feature vector corresponding to each target bounding box can refer to: performing person re-identification (REID) based on the target feature vector corresponding to each target bounding box to obtain the tracking and matching result corresponding to each target bounding box. Specifically, performing REID based on the target feature vector corresponding to each target bounding box to obtain the tracking and matching result corresponding to each target bounding box can include: determining the target historical trajectory matching the target object corresponding to the b-th target bounding box from multiple historical trajectories based on the target feature vector corresponding to the b-th target bounding box and the type of the b-th target bounding box, and using the target historical trajectory as the tracking and matching result of the target object corresponding to the b-th target bounding box. It should be understood that the above is only one possible example of obtaining the tracking and matching result, and other methods can be used in actual processing, which are not exhaustively listed here. Here, b is a positive integer, and the b-th target bounding box is any one of all target bounding boxes. Since the same processing is performed for each target bounding box, it will not be described in detail.

[0160] Combination Figure 5 The above image processing method is illustrated by way of example. A target image 50 is input into a target model 501 to obtain a 1 / 8 feature map 502 of the target image output by the target model 501, and a reconstructed image 503 of the target image output by the target model 501 is obtained. The reconstructed image 503 of the target image is then detected to obtain one or more target bounding boxes contained within the reconstructed image. Figure 5It can be seen that the size of the reconstructed image 503 of the target image is the same as that of the target image 50. The detection of the reconstructed image 503 to obtain one or more target bounding boxes contained within it can be achieved by inputting the reconstructed image 503 into a detection model 505, obtaining one or more target bounding boxes output by the detection model 505, and the type corresponding to each of these bounding boxes. The target detection model can be set according to actual conditions; for example, it can use Convolutional Neural Networks (CNN) or Spatial Transformer Networks (STNs). This embodiment does not exhaustively list all examples. For instance, in... Figure 5 The diagram illustrates any one of the target bounding boxes 5031 contained in the reconstructed image of the target image. In addition, the type corresponding to each target bounding box in the one or more target bounding boxes can also be obtained.

[0161] Furthermore, continue to combine Figure 5 For example, based on the center point coordinates 50311 of the target bounding box 5031 in the reconstructed image of the target image, the position point 5021 corresponding to the center point coordinates 50311 of the target bounding box 5031 in the feature map 502 of the target image is determined, and the feature vector of the position point 5021 in the feature map of the target image is extracted as the target feature vector corresponding to the target bounding box. Then, based on the target feature vector corresponding to the target bounding box 5031, person re-identification (REID) 504 is performed to obtain the tracking and matching result corresponding to the target bounding box. Here, REID can be implemented using a relevant model capable of REID processing, or it can be implemented using a preset REID relevant algorithm. This embodiment does not limit it. Since the same processing is performed for each target bounding box, it will not be described in detail.

[0162] As can be seen, by adopting the above scheme, the target bounding box can be obtained by detecting the reconstructed image of the target image, thus providing an accurate positional reference for subsequent tracking and matching, and ensuring the accuracy of the results obtained by tracking and matching through feature maps. Furthermore, by combining the target bounding boxes obtained from the reconstructed image of the target image with the feature maps, the target feature vectors corresponding to each target bounding box are determined, thereby realizing the tracking and matching process and obtaining the tracking and matching results. In this way, the computational load of subsequent tracking and matching processes can be reduced, and the processing efficiency can be improved. In addition, based on the description of the first aspect embodiment above, it can be seen that the reconstructed image of the target image can be a defect-free and complete image reconstructed from the original damaged, noisy, and incomplete target image. Therefore, detecting the target bounding box based on the reconstructed image of the target image can ensure the accuracy of the bounding box detection results.

[0163] A third aspect of this disclosure provides a model training apparatus, such as... Figure 6 As shown, it includes:

[0164] The feature extraction module 601 is used to input a sample image into a preset model to obtain a feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image; and to input the image within each of one or more bounding boxes contained in the sample image into the preset model to obtain a feature vector corresponding to each bounding box output by the second intermediate layer of the preset model.

[0165] The target loss function generation module 602 is used to generate a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box;

[0166] The target model update module 603 is used to update the preset model based on the target loss function. When the preset model meets the convergence condition, a target model is obtained. The target model is used to process the target image to obtain a feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching.

[0167] The target loss function generation module is used to, in the process of updating the consistency loss function for the i-th time, determine the i-th position in the feature map of the sample image based on the center point position of the i-th bounding box, where i is a positive integer; obtain the feature vector to be compared at the i-th position from the feature map of the sample image; obtain the vector difference corresponding to the i-th position based on the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position; obtain the consistency loss function updated for the i-th time based on the vector difference corresponding to the i-th position and the consistency loss function updated for the (i-1)-th time; and, if the process of updating the consistency loss function for the i-th time is the last process of updating the consistency loss function, obtain the target consistency loss function based on the consistency loss function updated for the i-th time, and obtain the target loss function based on the target consistency loss function.

[0168] The target loss function generation module is used to generate a first target contrast loss function based on the feature vectors corresponding to different bounding boxes; and to obtain the target loss function based on the target consistency loss function and the first target contrast loss function.

[0169] The feature extraction module is used to input the image within each first bounding box of one or more first bounding boxes of the first enhanced image and the image within each second bounding box of one or more second bounding boxes of the second enhanced image into the preset model, respectively, to obtain the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model, wherein the first enhanced image and the second enhanced image are obtained by processing the sample image using different enhancement methods;

[0170] The target loss function generation module is used to obtain a cyclic loss function based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box; and to obtain the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function.

[0171] The target loss function generation module is used to select the k-th first bounding box and the j-th first bounding box, and to select the k-th second bounding box and the j-th second bounding box, in the process of calculating the cumulative sum of similarity differences in the m-th calculation; wherein, the position of the k-th first bounding box in the first enhanced image is the same as the position of the k-th second bounding box in the second enhanced image, and the position of the j-th first bounding box in the first enhanced image is the same as the position of the j-th second bounding box in the second enhanced image, k and j are different, and m, k, and j are all positive integers; based on the first feature vector corresponding to the k-th first bounding box. The first similarity is obtained by taking the second feature vector corresponding to the j-th second bounding box and the second feature vector corresponding to the k-th second bounding box; the m-th similarity difference between the first similarity and the second similarity is calculated; the m-th cumulative similarity difference is obtained based on the m-th similarity difference and the cumulative sum of the (m-1)-th similarity difference; if the m-th calculation of the cumulative similarity difference is the last calculation of the cumulative similarity difference, the cyclic loss function is obtained based on the cumulative sum of the m-th similarity difference.

[0172] The target loss function generation module is used to obtain a second target contrast loss function based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box; obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function includes: obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, and the second target contrast loss function.

[0173] The target loss function generation module is used to select the nth bounding box, the nth first bounding box, and the nth second bounding box in the nth generation of the positive sample contrast loss function; where n is a positive integer, and the positions of the nth bounding box in the sample image, the nth first bounding box in the first enhanced image, and the nth second bounding box in the second enhanced image are the same; calculate the first positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box, and calculate the second positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the second feature vector corresponding to the nth second bounding box. The second positive sample contrast loss is calculated based on the feature vector corresponding to the nth first bounding box and the second feature vector corresponding to the nth second bounding box. Based on the first positive sample contrast loss, the second positive sample contrast loss, and the third positive sample contrast loss, a current positive sample contrast loss sub-function is obtained. The current positive sample contrast loss sub-function is added to the (n-1)th positive sample contrast loss function to obtain the nth positive sample contrast loss function. If the nth generation of the positive sample contrast loss function is the last generation of the positive sample contrast loss function, the second target contrast loss function is generated based on the nth positive sample contrast loss function. The feature extraction module is used to obtain the reconstructed images of the first enhanced image and the second enhanced image respectively output by the output layer of the preset model.

[0174] The target loss function generation module is used to obtain a first reconstruction loss sub-function based on the sample image and the reconstructed image of the first enhanced image, and to obtain a second reconstruction loss sub-function based on the sample image and the reconstructed image of the second enhanced image; to obtain a reconstruction loss function based on the first reconstruction loss sub-function and the second reconstruction loss sub-function; and to obtain the target loss function based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, the second target contrast loss function, and the reconstruction loss function.

[0175] The fourth aspect of this disclosure provides an image processing apparatus, such as... Figure 7 As shown, it includes:

[0176] Image processing module 701 is used to input a target image into a target model to obtain a feature map of the target image output by the target model, wherein the size of the feature map of the target image is smaller than the target image, the feature map of the target image is used for tracking and matching, and the target model is obtained based on the above model training method.

[0177] exist Figure 7 On the basis of, such as Figure 8As shown, the device further includes: a detection module 801, used to detect the reconstructed image of the target image to obtain one or more target bounding boxes contained in the reconstructed image of the target image; a tracking and matching module 802, used to determine the center point coordinates of each target bounding box in the one or more target bounding boxes; extract the target feature vector corresponding to each target bounding box from the feature map of the target image based on the center point coordinates of each target bounding box; obtain the tracking and matching result corresponding to each target bounding box based on the target feature vector corresponding to each target bounding box; and an image processing module 701, used to obtain the reconstructed image of the target image output by the target model.

[0178] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0179] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0180] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0181] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 1200. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0182] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0183] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above. For example, in some embodiments, the various methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the various methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the various methods described above by any other suitable means (e.g., by means of firmware).

[0184] According to yet another embodiment of the present disclosure, a vehicle is provided that includes the electronic equipment 900 described above.

[0185] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0189] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0190] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0191] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0192] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model training method, comprising: A sample image is input into a preset model to obtain a feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image. The image within each of one or more bounding boxes contained in the sample image is input into a preset model to obtain the feature vector corresponding to each bounding box output by the second intermediate layer of the preset model; Based on the feature map of the sample image and the feature vector corresponding to each bounding box, a target loss function is generated. The preset model is updated based on the target loss function. When the preset model meets the convergence condition, a target model is obtained. The target model is used to process the target image to obtain a feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching. The step of generating a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box includes: in the process of updating the consistency loss function in the i-th time, determining the i-th position in the feature map of the sample image based on the center point position of the i-th bounding box, where i is a positive integer; obtaining the feature vector to be compared at the i-th position from the feature map of the sample image; obtaining the vector difference corresponding to the i-th position based on the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position; obtaining the consistency loss function updated in the i-th time based on the vector difference corresponding to the i-th position and the consistency loss function updated in the (i-1)-th time; when the process of updating the consistency loss function in the i-th time is the last process of updating the consistency loss function, obtaining the target consistency loss function based on the consistency loss function updated in the i-th time, and obtaining the target loss function based on the target consistency loss function.

2. The method of claim 1, further comprising: Based on the feature vectors corresponding to different bounding boxes, a first target contrast loss function is generated; The step of obtaining the target loss function based on the target consistency loss function includes: obtaining the target loss function based on the target consistency loss function and the first target comparison loss function.

3. The method of claim 2, further comprising: The images within each first bounding box of one or more first bounding boxes of the first enhanced image and the images within each second bounding box of one or more second bounding boxes of the second enhanced image are respectively input into the preset model to obtain the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model. The first enhanced image and the second enhanced image are obtained by processing the sample image using different enhancement methods. Based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box, a cyclic loss function is obtained. The step of obtaining the target loss function based on the target consistency loss function and the first target comparison loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, and the cyclic loss function.

4. The method of claim 3, wherein, The recurrent loss function is obtained based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box, including: In the m-th calculation of the cumulative sum of similarity differences, the k-th first bounding box and the j-th first bounding box are selected, and the k-th second bounding box and the j-th second bounding box are also selected; wherein, the position of the k-th first bounding box in the first enhanced image is the same as the position of the k-th second bounding box in the second enhanced image, and the position of the j-th first bounding box in the first enhanced image is the same as the position of the j-th second bounding box in the second enhanced image, k and j are different, and m, k and j are all positive integers; A first similarity is obtained based on the first feature vector corresponding to the kth first bounding box and the second feature vector corresponding to the jth second bounding box, and a second similarity is obtained based on the first feature vector corresponding to the jth first bounding box and the second feature vector corresponding to the kth second bounding box. Calculate the m-th similarity difference between the first similarity and the second similarity; Based on the sum of the m-th similarity difference and the (m-1)-th similarity difference, the sum of the m-th similarity differences is obtained; If the process of calculating the cumulative sum of similarity differences in the mth time is the last process of calculating the cumulative sum of similarity differences, the cyclic loss function is obtained based on the cumulative sum of similarity differences in the mth time.

5. The method of claim 3 or 4, further comprising: Based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box, a second target contrast loss function is obtained; The step of obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, and the cyclic loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, and the second target comparison loss function.

6. The method according to claim 5, wherein, The second target contrast loss function is obtained based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box, including: In the nth generation of the positive sample contrast loss function, the nth bounding box, the nth first bounding box, and the nth second bounding box are selected; where n is a positive integer, and the positions of the nth bounding box in the sample image, the nth first bounding box in the first enhanced image, and the nth second bounding box in the second enhanced image are the same; A first positive sample contrast loss is calculated based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box; a second positive sample contrast loss is calculated based on the feature vector corresponding to the nth bounding box and the second feature vector corresponding to the nth second bounding box; and a third positive sample contrast loss is calculated based on the feature vector corresponding to the nth first bounding box and the second feature vector corresponding to the nth second bounding box. Based on the first positive sample contrast loss, the second positive sample contrast loss, and the third positive sample contrast loss, the current positive sample contrast loss sub-function is obtained; The nth positive sample contrast loss function is obtained by adding the current positive sample contrast loss function to the (n-1)th positive sample contrast loss function. If the nth generation positive sample contrast loss function is the same as the last generation positive sample contrast loss function, the second target contrast loss function is generated based on the nth positive sample contrast loss function.

7. The method according to claim 5, wherein, After inputting the first enhanced image and the second enhanced image into the preset model, the method further includes: obtaining the reconstructed images of the first enhanced image and the second enhanced image respectively output by the output layer of the preset model; obtaining a first reconstruction loss function based on the sample image and the reconstructed image of the first enhanced image; obtaining a second reconstruction loss function based on the sample image and the reconstructed image of the second enhanced image; and obtaining a reconstruction loss function based on the first reconstruction loss function and the second reconstruction loss function. The step of obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, and the second target comparison loss function includes: obtaining the target loss function based on the target consistency loss function, the first target comparison loss function, the cyclic loss function, the second target comparison loss function, and the reconstruction loss function.

8. An image processing method, comprising: The target image is input into the target model to obtain the feature map of the target image output by the target model, wherein the size of the feature map of the target image is smaller than the target image, and the feature map of the target image is used for tracking and matching, and the target model is obtained based on the method of any one of claims 1 to 7.

9. The method according to claim 8, wherein, After inputting the target image into the target model, the method further includes: obtaining a reconstructed image of the target image output by the target model; and detecting one or more target bounding boxes contained in the reconstructed image of the target image. After obtaining the feature map of the target image output by the target model, the method further includes: determining the center point coordinates of each target bounding box in the one or more target bounding boxes; extracting the target feature vector corresponding to each target bounding box from the feature map of the target image based on the center point coordinates of each target bounding box; and obtaining the tracking matching result corresponding to each target bounding box based on the target feature vector corresponding to each target bounding box.

10. A model training device, comprising: The feature extraction module is used to input a sample image into a preset model to obtain a feature map of the sample image output by the first intermediate layer of the preset model, wherein the size of the feature map of the sample image is smaller than that of the sample image; and to input the image within each of one or more bounding boxes contained in the sample image into the preset model to obtain a feature vector corresponding to each bounding box output by the second intermediate layer of the preset model. The target loss function generation module is used to generate a target loss function based on the feature map of the sample image and the feature vector corresponding to each bounding box; The target model update module is used to update the preset model based on the target loss function. When the preset model meets the convergence condition, a target model is obtained. The target model is used to process the target image to obtain a feature map of the target image. The size of the feature map of the target image is smaller than that of the target image. The feature map of the target image is used for tracking and matching. The target loss function generation module is used to, in the process of updating the consistency loss function for the i-th time, determine the i-th position in the feature map of the sample image based on the center point position of the i-th bounding box, where i is a positive integer; obtain the feature vector to be compared at the i-th position from the feature map of the sample image; obtain the vector difference corresponding to the i-th position based on the feature vector corresponding to the i-th bounding box and the feature vector to be compared at the i-th position; obtain the consistency loss function updated for the i-th time based on the vector difference corresponding to the i-th position and the consistency loss function updated for the (i-1)-th time; and, if the process of updating the consistency loss function for the i-th time is the last process of updating the consistency loss function, obtain the target consistency loss function based on the consistency loss function updated for the i-th time, and obtain the target loss function based on the target consistency loss function.

11. The apparatus according to claim 10, wherein, The target loss function generation module is used to generate a first target contrast loss function based on the feature vectors corresponding to different bounding boxes; and to obtain the target loss function based on the target consistency loss function and the first target contrast loss function.

12. The apparatus according to claim 11, wherein, The feature extraction module is used to input the image within each first bounding box of one or more first bounding boxes of the first enhanced image and the image within each second bounding box of one or more second bounding boxes of the second enhanced image into the preset model, respectively, to obtain the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box output by the second intermediate layer of the preset model, wherein the first enhanced image and the second enhanced image are obtained by processing the sample image using different enhancement methods; The target loss function generation module is used to obtain a cyclic loss function based on the first feature vector corresponding to each first bounding box and the second feature vector corresponding to each second bounding box; and to obtain the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function.

13. The apparatus according to claim 12, wherein, The target loss function generation module is used to select the k-th first bounding box and the j-th first bounding box, and to select the k-th second bounding box and the j-th second bounding box, in the process of calculating the cumulative sum of similarity differences in the m-th calculation; wherein, the position of the k-th first bounding box in the first enhanced image is the same as the position of the k-th second bounding box in the second enhanced image, and the position of the j-th first bounding box in the first enhanced image is the same as the position of the j-th second bounding box in the second enhanced image, k and j are different, and m, k, and j are all positive integers; based on the first feature vector corresponding to the k-th first bounding box. The first similarity is obtained by taking the second feature vector corresponding to the j-th second bounding box and the second feature vector corresponding to the k-th second bounding box; the m-th similarity difference between the first similarity and the second similarity is calculated; the m-th cumulative similarity difference is obtained based on the m-th similarity difference and the cumulative sum of the (m-1)-th similarity difference; if the m-th calculation of the cumulative similarity difference is the last calculation of the cumulative similarity difference, the cyclic loss function is obtained based on the cumulative sum of the m-th similarity difference.

14. The apparatus according to claim 12 or 13, wherein, The target loss function generation module is used to obtain a second target contrast loss function based on the feature vector corresponding to each bounding box, the first feature vector corresponding to each first bounding box, and the second feature vector corresponding to each second bounding box; obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, and the cyclic loss function includes: obtaining the target loss function based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, and the second target contrast loss function.

15. The apparatus according to claim 14, wherein, The target loss function generation module is used to select the nth bounding box, the nth first bounding box, and the nth second bounding box in the nth generation of the positive sample contrast loss function; where n is a positive integer, and the positions of the nth bounding box in the sample image, the nth first bounding box in the first enhanced image, and the nth second bounding box in the second enhanced image are the same; calculate the first positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the first feature vector corresponding to the nth first bounding box, and calculate the second positive sample contrast loss based on the feature vector corresponding to the nth bounding box and the second feature vector corresponding to the nth second bounding box. The second positive sample contrast loss is calculated based on the feature vector corresponding to the nth first bounding box and the second feature vector corresponding to the nth second bounding box. Based on the first positive sample contrast loss, the second positive sample contrast loss, and the third positive sample contrast loss, a current positive sample contrast loss sub-function is obtained. The current positive sample contrast loss sub-function is added to the (n-1)th positive sample contrast loss function to obtain the nth positive sample contrast loss function. If the nth generation of the positive sample contrast loss function is the last generation of the positive sample contrast loss function, the second target contrast loss function is generated based on the nth positive sample contrast loss function.

16. The apparatus according to claim 14, wherein, The feature extraction module is used to obtain the reconstructed images of the first enhanced image and the second enhanced image respectively output by the output layer of the preset model; The target loss function generation module is used to obtain a first reconstruction loss sub-function based on the sample image and the reconstructed image of the first enhanced image, and to obtain a second reconstruction loss sub-function based on the sample image and the reconstructed image of the second enhanced image; to obtain a reconstruction loss function based on the first reconstruction loss sub-function and the second reconstruction loss sub-function; and to obtain the target loss function based on the target consistency loss function, the first target contrast loss function, the cyclic loss function, the second target contrast loss function, and the reconstruction loss function.

17. An image processing apparatus, comprising: An image processing module is used to input a target image into a target model to obtain a feature map of the target image output by the target model, wherein the size of the feature map of the target image is smaller than the target image, the feature map of the target image is used for tracking and matching, and the target model is obtained based on the method of any one of claims 1 to 7.

18. The apparatus of claim 17, further comprising: The detection module is used to detect the reconstructed image of the target image and obtain one or more target bounding boxes contained in the reconstructed image of the target image; The tracking and matching module is used to determine the center point coordinates of each target bounding box within the one or more target bounding boxes; based on the center point coordinates of each target bounding box, extract the target feature vector corresponding to each target bounding box from the feature map of the target image; and based on the target feature vector corresponding to each target bounding box, obtain the tracking and matching result corresponding to each target bounding box. The image processing module is used to obtain the reconstructed image of the target image output by the target model.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7 or 8-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7 or 8-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7 or 8-9.

Images