Training method of image processing model, image processing method and device

Abstract

The present disclosure provides a training method of an image processing model, and relates to the technical fields of artificial intelligence, deep learning, computer vision and image processing. The specific implementation scheme is: initializing parameters of at least one of a plurality of to-be-trained modules according to parameters of a pre-trained model to obtain a to-be-trained model; inputting sample images into the pre-trained model and the to-be-trained model respectively to obtain a first output of the pre-trained model and a second output of the to-be-trained model; determining a loss of the to-be-trained model according to the first output and the second output; and adjusting parameters of the remaining to-be-trained modules except the at least one of the plurality of to-be-trained modules according to the loss, and returning to the step of inputting the sample images into the pre-trained model and the to-be-trained model respectively until the to-be-trained model meets a preset condition to obtain a trained image processing model. The present disclosure also provides an image processing method and device, an electronic device and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of deep learning, computer vision, and image processing technology. More specifically, this disclosure provides a method for training an image processing model, an image processing method, an apparatus, an electronic device, and a storage medium. Background Technology

[0002] Knowledge distillation is a commonly used model compression method. Model training methods based on knowledge distillation refer to using the training results of a large, highly accurate model to guide the training process of a smaller model, thereby enabling the smaller model to achieve similar accuracy metrics to the large model, thus achieving the effect of model compression. Summary of the Invention

[0003] This disclosure provides a training method for an image processing model, an image processing method, an apparatus, an electronic device, and a storage medium.

[0004] According to a first aspect, a training method for an image processing model is provided, the image processing model including multiple training modules; the method includes: initializing the parameters of at least one training module among the multiple training modules according to the parameters of a pre-trained model to obtain a training model; inputting sample images into the pre-trained model and the training model respectively to obtain a first output of the pre-trained model and a second output of the training model; determining the loss of the training model according to the first output and the second output; and adjusting the parameters of the remaining training modules among the multiple training modules except for at least one training module according to the loss, and returning to the step of inputting sample images into the pre-trained model and the training model respectively, until the training model meets preset conditions to obtain a trained image processing model.

[0005] According to the second aspect, an image processing method is provided, the method comprising: acquiring an image to be processed, wherein the image to be processed includes a query object; and inputting the image to be processed into a trained image processing model to obtain the location information of the query object; wherein the trained image processing model is trained according to the training method of the image processing model described above.

[0006] According to a third aspect, a training apparatus for an image processing model is provided, the image processing model including multiple training modules; the apparatus includes: an initialization module, used to initialize the parameters of at least one of the multiple training modules according to the parameters of a pre-trained model, to obtain a training model; a first processing module, used to input sample images into the pre-trained model and the training model respectively, to obtain a first output of the pre-trained model and a second output of the training model; a determination module, used to determine the loss of the training model according to the first output and the second output; and an adjustment module, used to adjust the parameters of the remaining training modules (excluding at least one training module) according to the loss, and return to the step of inputting sample images into the pre-trained model and the training model respectively, until the training model meets preset conditions, to obtain a trained image processing model.

[0007] According to a fourth aspect, an image processing apparatus is provided, the apparatus comprising: an acquisition module for acquiring an image to be processed, wherein the image to be processed includes a query object; and a second processing module for inputting the image to be processed into a trained image processing model to obtain location information of the query object; wherein the trained image processing model is trained using a training device for the image processing model described above.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods provided in this disclosure.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method provided in this disclosure when executed by a processor.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1This is an exemplary system architecture diagram of a training method for an image processing model and an image processing method applicable according to an embodiment of the present disclosure;

[0014] Figure 2 This is a flowchart of a training method for an image processing model according to an embodiment of the present disclosure;

[0015] Figure 3 This is an overall architecture diagram of a training method for an image processing model according to an embodiment of the present disclosure;

[0016] Figure 4 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0017] Figure 5 This is a block diagram of a training apparatus for an image processing model according to an embodiment of the present disclosure;

[0018] Figure 6 This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure;

[0019] Figure 7 This is a block diagram of an electronic device for training an image processing model and / or an image processing method according to an embodiment of the present disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] The DETR (Detection Transformer) model is a Transformer-based image processing model, specifically an object detection model. DETR can predict multiple objects simultaneously, matching the predicted results one-to-one with the ground truth values ​​using a binary matching method, directly outputting the bounding boxes of the object objects. Therefore, no post-processing operations such as deduplication of the network output are required. Thus, the DETR model does not require any post-processing modules, achieving true end-to-end output and avoiding the impact of post-processing modules on model performance.

[0022] However, DETR requires 10 to 20 times more training epochs than mainstream convolutional neural network detectors to converge. Furthermore, to achieve higher accuracy, DETR requires a larger number of parameters and datasets. In object detection applications such as autonomous driving and security monitoring, it is necessary to efficiently transfer knowledge from large models to smaller models.

[0023] Knowledge distillation-based model training methods enable the efficient transfer of knowledge from large models to smaller models. For example, in autonomous driving applications, a high-precision obstacle detection model can be trained using a massive dataset. Then, through knowledge distillation, a lightweight, high-precision smaller model can be distilled from the large model and deployed in automotive chips. Similarly, in security monitoring applications, a high-precision pedestrian detection model can be trained using a massive dataset. Then, through knowledge distillation, a lightweight, high-precision smaller model can be deployed in surveillance camera chips.

[0024] Larger models, with more parameters, can be called teacher models. Smaller models, with fewer parameters, can be called student models. Model training methods based on knowledge distillation are also called model distillation methods. Model distillation methods mainly include the following:

[0025] A response-based model distillation method calculates distillation loss based on the outputs of the teacher model (e.g., position and category) and the outputs of the student model (e.g., position and category).

[0026] A feature-based model distillation method calculates the distillation loss based on the output feature maps of the intermediate layers of the teacher model and the student model.

[0027] A relation-based model distillation method calculates the distillation loss based on the relation matrix between the outputs of different layers of the teacher model and the relation matrix between the outputs of different layers of the student model.

[0028] The models obtained by the above-mentioned model distillation methods can output relatively accurate results for simpler classification and recognition tasks. However, when the model is applied to more complex target detection tasks, the accuracy of the output results often fails to meet expectations.

[0029] Furthermore, existing distillation methods for object detection models primarily target convolutional neural network architectures, such as distilling the output of convolutional layers. However, for Transformer-based DETR object detection models, distilling the output of convolutional layers does not achieve satisfactory results, leading to poor object detection performance in the distilled model.

[0030] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0032] Figure 1 This is a schematic diagram of an exemplary system architecture according to an embodiment of the present disclosure, illustrating a training method for an image processing model and an image processing method to be applied. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, etc.

[0035] The image processing model training method and at least one of the image processing methods provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the image processing model training apparatus and image processing apparatus provided in this disclosure embodiment can generally be located in server 105. The image processing model training method and image processing method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the image processing model training apparatus and image processing apparatus provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0036] Figure 2 This is a flowchart of a training method for an image processing model according to an embodiment of the present disclosure.

[0037] The image processing model in this embodiment can be a small-scale student model. This embodiment also includes a pre-trained model, which can be a high-precision teacher model trained to full convergence. The teacher model and the student model have similar structures, both of which can be DETR model structures.

[0038] The DETR model structure can include a convolutional module, an encoding module, and a decoding module. The convolutional module extracts features from the image to obtain a feature map. The encoding and decoding modules form a Transformer structure. The encoding module converts the feature map into a feature sequence and encodes it based on the positional relationships between features, resulting in an encoded feature sequence. The decoding module uses a set of object lookup parameters (or object lookup vectors) to retrieve the location of the target object from the encoded feature sequence, obtaining the bounding box of the target object.

[0039] The convolutional, encoding, and decoding modules in the teacher model can be used as pre-trained modules, all of which are trained to full convergence. The convolutional, encoding, and decoding modules in the student model can be used as modules to be trained.

[0040] like Figure 2 As shown, the training method 200 of the image processing model includes operations S210 to S250.

[0041] In operation S210, the parameters of at least one of the multiple trainable modules are initialized according to the parameters of the pre-trained model to obtain the trainable model.

[0042] For example, the parameters of the decoding module to be trained in the student model can be initialized to be consistent with the parameters of the pre-trained decoding module in the teacher model, allowing the two models to share parameters. Since the decoding module has the strongest object query capability in the DETR model structure, sharing parameters between the student and teacher models enables the student model to possess the same object-finding ability as the teacher model, i.e., object detection capability.

[0043] The initialization of the trainable convolutional and encoding modules of the student model can adopt conventional initialization methods, such as initializing the parameters of the trainable convolutional and encoding modules according to a Gaussian distribution.

[0044] The initialized student model is used as the model to be trained (hereinafter referred to as the student model).

[0045] In operation S220, the sample images are input into the pre-trained model and the model to be trained, respectively, to obtain the first output of the pre-trained model and the second output of the model to be trained.

[0046] For example, the sample image can be an image that has undergone data augmentation processing such as scaling, cropping, and rotation. The sample image is input into the teacher model and the student model respectively, resulting in the first output of the teacher model and the second output of the student model.

[0047] The first output and the second output can be the final outputs of the teacher model and the student model, respectively. For example, the first output includes the location and category of the target object in the sample image predicted by the teacher model, and the second output includes the location and category of the target object in the sample image predicted by the student model.

[0048] The first output may also include the outputs of the various pre-trained modules of the teacher model, such as at least one of the outputs of the pre-trained convolutional module, the pre-trained encoding module, and the pre-trained decoding module.

[0049] Similarly, the second output may also include the outputs of the various trainable modules of the student model, such as at least one of the outputs of the trainable convolutional module, the trainable encoding module, and the trainable decoding module.

[0050] In operation S230, the loss of the model to be trained is determined based on the first and second outputs.

[0051] For example, distillation loss can be determined based on the difference between the final output of the teacher model and the final output of the student model. Truth loss can also be determined based on the difference between the final output of the student model and the ground truth (the true location and class). The overall loss is determined based on the distillation loss and the truth loss (e.g., their sum).

[0052] The distillation loss mentioned above can also include the difference between the output of the module to be trained and the output of the corresponding pre-trained module. For example, it includes the difference between the output of the convolutional module to be trained and the output of the pre-trained convolutional module, the difference between the output of the encoding module to be trained and the output of the pre-trained encoding module, and the difference between the output of the decoding module to be trained and the output of the pre-trained decoding module.

[0053] The differences mentioned above may include differences in mean squared error, cross-entropy, etc.

[0054] In operation S240, the parameters of the remaining training modules (excluding at least one) are adjusted according to the loss.

[0055] In operation S250, determine whether the model to be trained meets the preset conditions. If it does, training is complete; otherwise, return to operation S220.

[0056] For example, based on the overall loss of the student model, the gradients of the parameters of each module to be trained in the student model are calculated, and then gradient backpropagation is performed to update the parameters of each module to be trained.

[0057] It should be noted that, in order for the student model to retain the teacher model's ability to find objects, the decoding module to be trained does not need to participate in gradient backpropagation and parameter updates. Therefore, there is no need to calculate the gradient of the parameters of the decoding module to be trained.

[0058] Therefore, based on the overall loss of the student model, the gradients of the convolutional module and the encoding module to be trained can be calculated. Then, gradient backpropagation is performed to update the parameters of the convolutional module and the encoding module to be trained, thus obtaining the updated student model.

[0059] For the next sample image, return to operation S220 and repeat operations S220 to S240. For example, input the sample image into the teacher model and the updated student model respectively to obtain new first and second outputs. Calculate a new loss based on the new first and second outputs, and continue to update the parameters of the convolutional and encoding modules of the student model based on the new loss. Then return to operation S220 and repeat the above operations until the student model meets the preset conditions, resulting in a trained image processing model. Meeting the preset conditions may include student model convergence.

[0060] In each training round, the parameters of the decoding module to be trained in this trained image processing model remain consistent with those of the pre-trained decoding module and do not participate in parameter updates. This enables the image processing model to have the ability to find objects, thereby improving the object detection effect of the image processing model and thus improving the accuracy of object detection.

[0061] The embodiments of this disclosure initialize the parameters of at least one trainable module of the student model to be consistent with the parameters of the corresponding pre-trained module in the teacher model, and do not update the parameters of the at least one trainable module during gradient backpropagation, so that the student model has the target detection capability of the teacher model, thereby improving the target detection effect of the student model and thus improving the target detection accuracy.

[0062] According to embodiments of this disclosure, the pre-trained model includes a pre-trained decoding module, which includes M decoding layers. Multiple trainable modules include a trainable decoding module, which includes m decoding layers. M and m are both integers greater than 1, and M is greater than m. The above operation S210 includes: initializing the parameters of all m decoding layers of the trainable decoding module to be consistent with the parameters of the Mth decoding layer of the pre-trained decoding module; or initializing the parameters of the m decoding layers of the trainable decoding module to be consistent with the parameters of the last m decoding layers of the pre-trained decoding module.

[0063] The teacher model and the student model have the same structure. The difference is that the teacher model has more layers in each pre-training module than the student model has in each untrained module, and the teacher model can detect more objects than the student model.

[0064] For example, the teacher model's pre-trained convolutional module includes 10 convolutional layers, while the student model's untrained convolutional module includes 5 convolutional layers. The teacher model's pre-trained encoding module includes 6 encoding layers, while the student model's untrained encoding module includes 2 encoding layers. The teacher model's pre-trained decoding module includes 6 (M=6) decoding layers, while the student model's untrained decoding module includes 2 (m=2) decoding layers.

[0065] When initializing the decoding module to be trained, each layer of the decoding module to be trained can be initialized with the same parameters as the last layer of the pre-trained decoding module. Alternatively, the two decoding layers of the decoding module to be trained can be initialized with the same parameters as the last two decoding layers of the pre-trained decoding module.

[0066] Since the deeper the decoding module, the more important the features output by that layer are to the object detection result, and the more important the parameters of that layer become, the more important it is. Therefore, initializing the parameters of the decoding module to be trained based on the parameters of the last layer or the last n layers of the pre-trained decoding module can improve the decoding capability of the module to be trained.

[0067] According to embodiments of this disclosure, the sample image includes a target object, the parameters of the pre-trained model further include a first object lookup parameter, and the parameters of the model to be trained further include a second object lookup parameter. The first object lookup parameter corresponds to the location information of the target object. Operation S210 further includes initializing the second object lookup parameter to be consistent with the first object lookup parameter. Operation S240 further includes adjusting the remaining parameters in the model to be trained, except for the parameters of at least one training module and the second object lookup parameter.

[0068] In addition to the parameters of each pre-trained module, the teacher model's parameters also include the first object query parameters. Correspondingly, in addition to the parameters of each module to be trained, the student model's parameters also include the second object query parameters.

[0069] The object query parameter is a set of learnable parameters in DETR. It serves as input to the decoding module, enabling the encoding module to output the location information of the target object corresponding to the object query parameter.

[0070] The search for multiple target objects in the sample image is unordered. The correspondence between the first object search parameter of the teacher model and the position information of the target object is reflected in the fact that after the first search parameter and the parameters of the pre-trained decoder are fixed, the order in which the teacher model searches for objects is fixed. This is also a structural feature of DETR, known as the invariance of the decoder and object search parameters.

[0071] Therefore, when initializing the student model's parameters, the structural features of DETR—namely, the invariance of the decoder and object lookup parameters—can be utilized to initialize the student model's second object lookup parameters to be consistent with the teacher model's first lookup parameters. When adjusting the student model's parameters, the second object lookup parameters do not participate in the parameter adjustment; that is, only the parameters of the decoding module to be trained and the second object lookup parameters are adjusted. This ensures that the order in which the student model looks up objects is consistent with the order in which the teacher model looks up objects, thereby improving the student model's object detection capability.

[0072] According to embodiments of this disclosure, the pre-trained model includes multiple pre-training modules, each corresponding to a multiple trainable module. A first output includes the first output of each of the multiple pre-training modules, and a second output includes the second output of each of the multiple trainable modules. Operation S230 includes, for each trainable module, determining the loss of the trainable module based on its second output and the first output of the corresponding pre-training module; and determining the loss of the trainable model based on the losses of each of the multiple trainable modules.

[0073] The following is combined Figure 3 The overall architecture of the training method for the image processing model provided in this disclosure is described.

[0074] Figure 3 This is an overall architecture diagram of a training method for an image processing model according to an embodiment of the present disclosure.

[0075] like Figure 3 As shown, the image processing model in this embodiment can be a student model 320. This embodiment also includes a teacher model 310 that has been trained to full convergence. The teacher model 310 and the student model 320 have the same structure. For example, the teacher model 310 includes a pre-trained convolutional module 311, a pre-trained encoding module 312, and a pre-trained decoding module 313. The student model 320 includes a convolutional module 321 to be trained, an encoding module 322 to be trained, and a decoding module 323 to be trained.

[0076] The teacher model 310 also includes a first object lookup parameter 314, which serves as the input to the pre-trained decoding module 313, enabling the pre-trained decoding module 313 to output the first position information of the target object in the sample image 301.

[0077] The student model 320 also includes a second object lookup parameter 324, which serves as the input to the decoding module 323 to be trained, so that the decoding module 323 to be trained outputs the second position information of the target object in the sample image 301.

[0078] Before training, the parameters of the training decoding module 323 of the student model 320 are initialized to be consistent with the parameters of the pre-trained decoding module 313 of the teacher model 310, and the second object lookup parameter 324 of the student model 320 is initialized to be consistent with the first object lookup parameter 314 of the teacher model 310.

[0079] According to embodiments of this disclosure, for a convolutional module to be trained, the convolutional loss between the second output of the convolutional module to be trained and the first output of the pre-trained convolutional module is calculated to obtain the loss of the convolutional module to be trained; for an encoding module to be trained, the encoding loss between the second output of the encoding module to be trained and the first output of the pre-trained encoding module is calculated to obtain the loss of the encoding module to be trained; for a decoding module to be trained, the decoding loss between the second output of the decoding module to be trained and the first output of the pre-trained decoding module is calculated to obtain the loss of the decoding module to be trained.

[0080] After training begins, the sample image 301 is input into the teacher model 310, which yields the first output feat_t of the pre-trained convolutional module 311, the first output memory_t of the pre-trained encoding module 312, and the first output logit_t of the pre-trained decoding module 313.

[0081] When sample image 301 is input into student model 320, the second output feat_s of the convolutional module 321 to be trained, the second output memory_s of the encoding module 322 to be trained, and the second output logit_s of the decoding module 323 to be trained can be obtained.

[0082] Based on the first output feat_t of the pre-trained convolutional module 311 and the second output feat_s of the convolutional module 321 to be trained, the convolutional loss 331 can be calculated. For example, the mean square error (MSE) between the first output feat_t and the second output feat_s is calculated as the convolutional loss 331.

[0083] Based on the first output memory_t of the pre-trained encoding module 312 and the second output memory_s of the encoding module to be trained 322, the encoding loss 332 can be calculated. For example, the mean square error between the first output memory_t and the second output memory_s can be calculated as the encoding loss 332.

[0084] Based on the first output logit_t of the pre-trained decoding module 313 and the second output logit_s of the decoding module 323 to be trained, the decoding loss 333 can be calculated. For example, the information divergence (Kullback-Leibler Divergence, KL divergence) between the first output logit_t and the second output logit_s can be calculated as the decoding loss 333.

[0085] For example, adding the convolution loss 331, encoding loss 332, and decoding loss 333 yields the overall loss 334. The parameters of the trainable convolutional module 321 and trainable encoding module 322 of the student model 320 can be adjusted based on the overall loss 334. The parameters of the second object lookup parameter 324 and the trainable decoding module 323 are not involved in the parameter adjustment of the student model 320.

[0086] In this embodiment, during training, the parameters of the decoding module 323 to be trained in the student model 320 are kept consistent with the parameters of the pre-trained decoding module 313 in the teacher model 310, and the second object search parameter 324 of the student model 320 is kept consistent with the first object search parameter 314 of the teacher model 310. This enables the student model 320 to possess the object detection capability of the teacher model 310, thereby improving the object detection effect of the student model and thus improving the object detection accuracy.

[0087] According to embodiments of this disclosure, multiple pre-training modules include a pre-training encoding module, which includes N encoding layers; multiple trainable modules include a trainable encoding module, which includes n encoding layers, where N and n are both integers greater than 1, and N is greater than n; the above operation S230 includes, for the trainable encoding module, calculating encoding losses by comparing the first output of the Nth encoding layer of the pre-training encoding module with the second output of each encoding layer of the trainable encoding module, to obtain n encoding losses; and determining the loss of the trainable encoding module based on the n encoding losses.

[0088] The teacher model has a larger number of pre-trained encoding modules, for example, including 6 encoding layers (N=6). The student model has a smaller number of encoding modules to be trained, for example, including 2 encoding layers (n=2).

[0089] When calculating the encoding loss, the encoding loss can be calculated separately for the first output of the last layer of the pre-trained encoding module and the second output of each layer of the encoding module to be trained. This yields n encoding losses, and the sum of these n encoding losses can be used as the overall encoding loss.

[0090] For example, the first coding loss is obtained by calculating the mean square error between the first output of the sixth coding layer of the pre-trained coding module and the second output of the first coding layer of the coding module to be trained.

[0091] Then, the mean square error between the first output of the sixth coding layer of the pre-trained coding module and the second output of the second coding layer of the coding module to be trained is calculated to obtain the second coding loss.

[0092] Next, the first coding loss and the second coding loss are added together to obtain the overall coding loss.

[0093] Since the deeper the encoding module, the more important the output of that layer is to the object detection result, using the first output of the pre-trained encoding module to calculate the encoding loss with each layer of the encoding module to be trained can improve the accuracy of the encoding loss, thereby improving the training effect of the student model and the object detection effect of the image.

[0094] Figure 4 This is a flowchart of an image processing method according to an embodiment of the present disclosure.

[0095] like Figure 4 As shown, the image processing method 400 includes operations S410 to S420.

[0096] In operation S410, an image to be processed is acquired, which includes a query object. The query object can be an object, animal, person, etc., in the image. Multiple query objects may be included.

[0097] In operation S420, the image to be processed is input into the trained image processing model to obtain the location information of the object to be queried.

[0098] The trained image processing model can be a student model trained according to the training method of the above image processing model.

[0099] By inputting the image to be processed into a trained image processing model, the location information of the object to be queried in the image can be obtained. The location information can be the bounding box of the object to be queried.

[0100] This embodiment utilizes a trained image processing model to locate the target object in the image to be processed, thereby obtaining the location information of the target object. Since some parameters of the trained image processing model (such as the parameters of the decoding module and the object search parameters) are consistent with the corresponding parameters of the pre-trained model, the retrieval effect of the target object can be improved, and the accuracy of the location information of the target object can be improved.

[0101] Figure 5 This is a block diagram of a training apparatus for an image processing model according to an embodiment of the present disclosure.

[0102] like Figure 5As shown, the training device 500 for the image processing model includes an initialization module 501, a first processing module 502, a determination module 503, and an adjustment module 504.

[0103] The initialization module 501 is used to initialize the parameters of at least one of the multiple training modules according to the parameters of the pre-trained model, so as to obtain the training model.

[0104] The first processing module 502 is used to input sample images into the pre-trained model and the model to be trained, respectively, to obtain the first output of the pre-trained model and the second output of the model to be trained.

[0105] The determination module 503 is used to determine the loss of the model to be trained based on the first output and the second output.

[0106] The adjustment module 504 is used to adjust the parameters of the remaining training modules (excluding at least one) among the multiple training modules according to the loss, and return to the first processing module 502 until the training model meets the preset conditions, thus obtaining the trained image processing model.

[0107] According to embodiments of this disclosure, the pre-trained model includes a pre-trained decoding module, which includes M decoding layers. Multiple trainable modules include a trainable decoding module, which includes m decoding layers. Both M and m are integers greater than 1, and M is greater than m. The initialization module 501 includes one of a first initialization unit and a second initialization unit.

[0108] The first initialization unit is used to initialize the parameters of the m decoding layers of the decoding module to be trained to be consistent with the parameters of the Mth decoding layer of the pre-trained decoding module.

[0109] The second initialization unit is used to initialize the parameters of the m decoding layers of the decoding module to be trained to be consistent with the parameters of the last m decoding layers of the pre-trained decoding module.

[0110] The adjustment module 504 is used to adjust the parameters of the remaining training modules (excluding the decoding module) among the multiple training modules according to the loss.

[0111] According to embodiments of this disclosure, the sample image includes a target object, the parameters of the pre-trained model further include a first object lookup parameter, and the parameters of the model to be trained further include a second object lookup parameter. The first object lookup parameter corresponds to the location information of the target object. The initialization module 501 further includes a third initialization unit.

[0112] The third initialization unit is used to initialize the second object lookup parameters to be consistent with the first object lookup parameters.

[0113] The adjustment module 504 is also used to adjust the remaining parameters in the model to be trained, except for the parameters of at least one module to be trained and the second object lookup parameter.

[0114] According to embodiments of this disclosure, the pre-trained model includes multiple pre-training modules, each corresponding to a multiple module to be trained. The first output includes the first output of each of the multiple pre-training modules, and the second output includes the second output of each of the multiple modules to be trained. The determining module 503 includes a first determining unit and a second determining unit.

[0115] The first determining unit is used to determine the loss of each module to be trained based on the second output of the module to be trained and the first output of the corresponding pre-trained module.

[0116] The second determining unit is used to determine the loss of the model to be trained based on the individual losses of the multiple modules to be trained.

[0117] According to embodiments of this disclosure, multiple pre-trained modules include a pre-trained encoding module, which includes N encoding layers; multiple trainable modules include a trainable encoding module, which includes n encoding layers, where N and n are both integers greater than 1, and N is greater than n. A first determining unit is configured to, for each trainable encoding module, calculate encoding losses by comparing the first output of the Nth encoding layer of the pre-trained encoding module with the second output of each encoding layer of the trainable encoding module, obtaining n encoding losses; and determine the loss of the trainable encoding module based on the n encoding losses.

[0118] According to embodiments of this disclosure, multiple pre-trained modules include a pre-trained convolutional module, a pre-trained encoding module, and a pre-trained decoding module; multiple modules to be trained include a to-be-trained convolutional module, a to-be-trained encoding module, and a to-be-trained decoding module. The first determining unit includes a first computation unit, a second computation unit, and a third computation unit.

[0119] The first computational unit is used to calculate the convolution loss between the second output of the convolutional module to be trained and the first output of the pre-trained convolutional module for the convolutional module to be trained, so as to obtain the loss of the convolutional module to be trained.

[0120] The second computational unit is used to calculate the encoding loss between the second output of the encoding module to be trained and the first output of the pre-trained encoding module for the encoding module to be trained, so as to obtain the loss of the encoding module to be trained.

[0121] The third computational unit is used to calculate the decoding loss between the second output of the decoding module to be trained and the first output of the pre-trained decoding module for the decoding module to be trained, so as to obtain the loss of the decoding module to be trained.

[0122] According to embodiments of this disclosure, the sample image includes at least one target object, the first output includes first location information of at least one target object, and the second output includes second location information of at least one target object. Preset conditions include convergence of the model to be trained.

[0123] Figure 6 This is a block diagram of an image processing apparatus according to an embodiment of the present disclosure.

[0124] like Figure 6 As shown, the image processing device 600 includes an acquisition module 601 and a second processing module 602.

[0125] The acquisition module 601 is used to acquire the image to be processed, wherein the image to be processed includes the object to be queried.

[0126] The second processing module 602 is used to input the image to be processed into the trained image processing model to obtain the location information of the object to be queried.

[0127] The trained image processing model is obtained by training the image processing model using the training device described above.

[0128] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0129] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0130] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0131] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as image processing model training methods and / or image processing methods. For example, in some embodiments, the image processing model training methods and / or image processing methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the image processing model training methods and / or image processing methods described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured by any other suitable means (e.g., by means of firmware) to perform training methods and / or image processing methods for the image processing model.

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0138] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0139] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for an image processing model, wherein both the image processing model and the pre-trained model employ a DETR object detection structure, the image processing model includes a training convolutional module, a training encoder module, and a training decoder module, and the pre-trained model includes a pre-trained convolutional module, a pre-trained encoder module, and a pre-trained decoder module, the method comprising: The parameters of the decoding module to be trained are initialized to be consistent with the parameters of the pre-trained decoding module, and the second object lookup parameter, which is the input of the decoding module to be trained, is initialized to be consistent with the first object lookup parameter, which is the input of the pre-trained decoding module, so as to obtain the model to be trained. The first object lookup parameter indicates the order of object lookup from the encoded feature sequence. Initializing the parameters of the decoding module to be trained to be consistent with the parameters of the pre-trained decoding module includes: The parameters of the m decoding layers in the decoding module to be trained are initialized to be consistent with the parameters of the Mth decoding layer of the pre-trained decoding module, where the Mth decoding layer is the deepest decoding layer of the pre-trained decoding module, or... The parameters of the m decoding layers in the decoding module to be trained are initialized to be consistent with the parameters of the last m decoding layers of the pre-trained decoding module. The last m decoding layers are the m deepest decoding layers of the pre-trained decoding module, where M and m are both integers greater than 1, and M is greater than m. The sample images are input into the pre-trained model and the model to be trained, respectively, to obtain the first output of the pre-trained model and the second output of the model to be trained. Wherein, the first output includes the outputs of the convolutional module to be trained, the encoding module to be trained, and the decoding module to be trained, and the second output includes the outputs of the pre-trained convolutional module, the pre-trained encoding module, and the pre-trained decoding module. The output of the decoding module to be trained includes first position information output based on the first object lookup parameters, and the output of the pre-trained decoding module includes second position information output based on the second object lookup parameters. Based on the first output and the second output, determine the loss of the model to be trained; and Based on the loss, adjust the parameters other than the parameters of the decoding module to be trained and the second object lookup parameters, and return to the step of inputting the sample images into the pre-trained model and the model to be trained respectively, until the model to be trained meets the preset conditions, and obtain the trained image processing model.

2. The method according to claim 1, wherein, The sample image includes a target object, and the first object lookup parameter corresponds to the location information of the target object.

3. The method according to claim 1 or 2, wherein, The step of determining the loss of the model to be trained based on the first output and the second output includes: For each module to be trained, the loss of the module to be trained is determined based on the second output of the module to be trained and the first output of the corresponding pre-trained module; and The loss of the model to be trained is determined based on the losses of each of the multiple modules to be trained.

4. The method according to claim 3, wherein, The plurality of pre-trained modules include a pre-trained encoding module, the pre-trained encoding module includes N encoding layers, the plurality of modules to be trained include a module to be trained, the module to be trained includes n encoding layers, where N and n are both integers greater than 1, and N is greater than n; The step of determining the loss of each module to be trained based on its second output and the first output of the corresponding pre-trained module includes: for the encoding module to be trained, The first output of the Nth coding layer of the pre-trained coding module is compared with the second output of each coding layer of the coding module to be trained to calculate the coding loss, resulting in n coding losses. as well as The loss of the coding module to be trained is determined based on the n coding losses.

5. The method according to claim 3 or 4, wherein, Based on the second output of the module to be trained and the first output of the corresponding pre-trained module, the loss of the module to be trained is determined to include: For the convolutional module to be trained, calculate the convolutional loss between the second output of the convolutional module to be trained and the first output of the pre-trained convolutional module to obtain the loss of the convolutional module to be trained; For the coding module to be trained, the coding loss between the second output of the coding module to be trained and the first output of the pre-trained coding module is calculated to obtain the loss of the coding module to be trained; For the decoding module to be trained, the decoding loss between the second output of the decoding module to be trained and the first output of the pre-trained decoding module is calculated to obtain the loss of the decoding module to be trained.

6. The method according to any one of claims 1 to 5, wherein, The sample image includes at least one target object, the first output includes the first location information of the at least one target object, and the second output includes the second location information of the at least one target object.

7. The method according to any one of claims 1 to 6, wherein, The preset conditions include the convergence of the model to be trained.

8. An image processing method, comprising: Obtain the image to be processed, wherein the image to be processed includes the object to be queried; as well as The image to be processed is input into a trained image processing model to obtain the location information of the object to be queried; The trained image processing model is obtained by the method according to any one of claims 1 to 7.

9. A training apparatus for an image processing model, wherein both the image processing model and the pre-trained model employ a DETR object detection structure, the image processing model includes a training convolutional module, a training encoder module, and a training decoder module, and the pre-trained model includes a pre-trained convolutional module, a pre-trained encoder module, and a pre-trained decoder module, and the training apparatus comprises: An initialization module is used to initialize the parameters of the decoding module to be trained to be consistent with the parameters of the pre-trained decoding module, and to initialize the second object lookup parameter, which is the input of the decoding module to be trained, to be consistent with the first object lookup parameter, which is the input of the pre-trained decoding module, so as to obtain the model to be trained. The first object lookup parameter indicates the order of object lookup from the encoded feature sequence. The initialization module includes: One of a first initialization unit or a second initialization unit, wherein the first initialization unit is used to initialize the parameters of the m decoding layers included in the decoding module to be trained to be consistent with the parameters of the Mth decoding layer of the pre-trained decoding module, wherein the Mth decoding layer is the deepest decoding layer of the pre-trained decoding module; and the second initialization unit is used to initialize the parameters of the m decoding layers included in the decoding module to be trained to be consistent with the parameters of the last m decoding layers of the pre-trained decoding module, wherein the last m decoding layers are the deepest m decoding layers of the pre-trained decoding module, wherein M and m are both integers greater than 1, and M is greater than m. The third initialization unit is used to initialize the second object lookup parameter, which is the input to the decoding module to be trained, to be consistent with the first object lookup parameter, which is the input to the pre-trained decoding module. The first processing module is used to input sample images into the pre-trained model and the model to be trained, respectively, to obtain the first output of the pre-trained model and the second output of the model to be trained. Wherein, the first output includes the outputs of the convolutional module to be trained, the encoding module to be trained, and the decoding module to be trained, and the second output includes the outputs of the pre-trained convolutional module, the pre-trained encoding module, and the pre-trained decoding module. The output of the decoding module to be trained includes first position information output based on the first object lookup parameters, and the output of the pre-trained decoding module includes second position information output based on the second object lookup parameters. The determination module is configured to determine the loss of the model to be trained based on the first output and the second output; and The adjustment module is used to adjust the parameters other than the parameters of the decoding module to be trained and the second object lookup parameters according to the loss, and return to the step of inputting the sample images into the pre-trained model and the model to be trained respectively, until the model to be trained meets the preset conditions, and a trained image processing model is obtained.

10. The training apparatus according to claim 9, wherein, The sample image includes a target object, and the first object lookup parameter corresponds to the location information of the target object.

11. The training apparatus according to claim 9 or 10, wherein, The second output includes the second output of each of the plurality of trainable modules; the determining module includes: The first determining unit is configured to, for each trainable module, determine the loss of the trainable module based on the second output of the trainable module and the first output of the corresponding pre-trained module; and The second determining unit is used to determine the loss of the model to be trained based on the losses of each of the multiple modules to be trained.

12. The training apparatus according to claim 11, wherein, The plurality of pre-trained modules include a pre-trained encoding module, which includes N encoding layers. The plurality of modules to be trained include a module to be trained, which includes n encoding layers. N and n are both integers greater than 1, and N is greater than n. The first determining unit is configured to, for the module to be trained, calculate encoding losses by comparing the first output of the Nth encoding layer of the pre-trained encoding module with the second output of each encoding layer of the module to be trained, to obtain n encoding losses; and determine the loss of the module to be trained based on the n encoding losses.

13. The training apparatus according to claim 11 or 12, wherein, The first determining unit includes: The first computing unit is used to calculate the convolution loss between the second output of the convolution module to be trained and the first output of the pre-trained convolution module for the convolution module to be trained, so as to obtain the loss of the convolution module to be trained. The second calculation unit is used to calculate the encoding loss between the second output of the encoding module to be trained and the first output of the pre-trained encoding module for the encoding module to be trained, so as to obtain the loss of the encoding module to be trained. The third calculation unit is used to calculate the decoding loss between the second output of the decoding module to be trained and the first output of the pre-trained decoding module for the decoding module to be trained, so as to obtain the loss of the decoding module to be trained.

14. The training apparatus according to any one of claims 9 to 13, wherein, The sample image includes at least one target object, the first output includes the first location information of the at least one target object, and the second output includes the second location information of the at least one target object.

15. The training apparatus according to any one of claims 9 to 14, wherein, The preset conditions include the convergence of the model to be trained.

16. An image processing apparatus, comprising: The acquisition module is used to acquire the image to be processed, wherein the image to be processed includes the object to be queried; as well as The second processing module is used to input the image to be processed into a trained image processing model to obtain the location information of the object to be queried. The trained image processing model is obtained by the apparatus according to any one of claims 9 to 15.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 8 when executed by a processor.

Images