Model training method, image processing method, device, equipment and medium

Abstract

The present disclosure provides a model training method, an image processing method, an apparatus, a device and a medium, which can be applied to the field of artificial intelligence technology. The method comprises: in response to a first visual processing task for training a pre-trained model, obtaining a first visual processing task template; inputting the first visual processing task template and a sample image into the pre-trained model, processing the sample image and the first visual processing task template by using a first adapter to obtain a first visual processing result; training the pre-trained model by using the first visual processing result and a first visual processing label corresponding to the sample image to obtain a first visual processing task model, and the model parameters of a backbone network remain unchanged in the training process of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, and the label information is used to guide the pre-trained model to output the first visual processing result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to a model training method, an image processing method, an apparatus, a device, and a medium. BACKGROUND

[0002] For a new processing task, the model parameters usually need to be globally adjusted again, thereby consuming more resources, wasting more training time, and the model parameters obtained by training occupy more storage space. SUMMARY

[0003] In view of the above problems, the present disclosure provides a model training method, an image processing method, an apparatus, a device, and a medium.

[0004] According to a first aspect of the present disclosure, a model training method is provided, comprising: in response to a first visual processing task for training a pre-trained model, obtaining a first visual processing task template, wherein the pre-trained model comprises a backbone network and a first adapter corresponding to the first visual processing task, and the model parameters of the backbone network are obtained by pre-training; inputting the first visual processing task template and a sample image into the pre-trained model, processing the sample image and the first visual processing task template by using the first adapter to obtain a first visual processing result; training the pre-trained model by using the first visual processing result and a first visual processing label corresponding to the sample image to obtain a first visual processing task model, wherein the model parameters of the backbone network remain unchanged during the training of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task.

[0005] According to an embodiment of the present disclosure, before the first visual processing task template is obtained in response to the first visual processing task for training the pre-trained model, the model training method further comprises: obtaining a template image based on the first visual processing task; and obtaining the first visual processing task template and the first visual processing label based on the template image.

[0006] According to an embodiment of the present disclosure, the pre-trained model is trained by using the first visual processing result and the first visual processing label corresponding to the sample image to obtain the first visual processing task model, comprising: obtaining loss information according to the first visual processing result and the first visual processing label; performing reverse gradient optimization on parameter information of the first adapter based on the loss information until the loss information meets a preset condition to obtain the first visual processing task model.

[0007] According to an embodiment of the present disclosure, the backbone network of the pre-trained model comprises a target visual processing module and at least two target network blocks, the at least two target network blocks comprising a first target network block and a second target network block; a first adapter is arranged between the at least two target network blocks; the first visual processing task template and the sample image are input into the pre-trained model, and the sample image and the first visual processing task template are processed by using the first adapter to obtain a first visual processing result, comprising: inputting the sample image and the first visual processing task template into the first target network block to output a first feature map and a first template feature map; inputting the first feature map and the first template feature map into the first adapter to output a second feature map and a second template feature map; inputting the second feature map and the second template feature map into the second target network block to output a third feature map and a third template feature map; and inputting the third feature map and the third template feature map into the target visual processing module to output the first visual processing result.

[0008] According to an embodiment of the present disclosure, the above-mentioned model training method further comprises: using a trained second adapter corresponding to a second visual processing task to replace the first adapter in the first visual processing task model to obtain a second visual processing task model, wherein the second visual processing task model is used to process a second visual processing task of an image, and the trained second adapter is obtained by training the pre-trained model based on the second visual processing task.

[0009] According to an embodiment of the present disclosure, before the first visual processing task template and the sample image are input into the pre-trained model, and the sample image and the first visual processing task template are processed by using the first adapter to obtain the first visual processing result, the above-mentioned model training method further comprises: in response to receiving a sample acquisition instruction from the electronic device, determining a sample image from a sample database; and calling a sample transmission interface to acquire the sample image from the sample database.

[0010] The second aspect of the present disclosure provides an image processing method, comprising: obtaining a to-be-processed image; obtaining a first visual processing task model according to the above-mentioned model training method; inputting the to-be-processed image into the first visual processing task model to output a target visual processing result.

[0011] The third aspect of the present disclosure provides a model training apparatus, comprising: a first acquisition module configured to acquire a first visual processing task template in response to a first visual processing task for training a pre-trained model, wherein the pre-trained model comprises a backbone network and a first adapter corresponding to the first visual processing task, and model parameters of the backbone network are obtained through pre-training; a first input module configured to input the first visual processing task template and a sample image into the pre-trained model, process the sample image and the first visual processing task template by using the first adapter, and obtain a first visual processing result; and a first training module configured to train the pre-trained model by using the first visual processing result and a first visual processing label corresponding to the sample image, and obtain a first visual processing task model, wherein the model parameters of the backbone network remain unchanged during the training of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task.

[0012] The third aspect of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the above method.

[0013] The fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above method.

[0014] According to the model training method, the image processing method, the apparatus, the device and the medium provided by the present disclosure, by keeping the model parameters of the backbone network unchanged during the training of the pre-trained model and adjusting only the parameters of the first adapter, the fitting capability of the pre-trained model for the first visual processing task is enhanced through the adapter, global adjustment of the parameters of the pre-trained model is avoided, the memory required for storing the model parameters is reduced, the resources of the pre-trained model are fully utilized, the sample images required for the training process are saved, and then the first visual processing result is guided by the first visual processing task template, thereby combining the first adapter with the first visual processing task template, enhancing the generalization capability of the pre-trained model during the training process, achieving the training effect meeting the requirements under the condition of using only a small amount of sample images, obtaining the first visual processing task model meeting the requirements, saving the resources of the sample images, and improving the training efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0015] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0016] Figure 1 This illustration schematically depicts an application scenario of a model training method or image processing method according to embodiments of the present disclosure.

[0017] Figure 2 A flowchart illustrating a model training method according to an embodiment of the present disclosure is shown schematically.

[0018] Figure 3 A schematic diagram illustrating a first vision processing task model according to an embodiment of the present disclosure is shown.

[0019] Figure 4 A schematic diagram illustrating the acquisition of loss information according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A schematic diagram of a first visual processing result acquisition method according to an embodiment of the present disclosure is shown.

[0021] Figure 6 A schematic diagram illustrating the set of output feature maps according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A schematic diagram illustrating a training process according to an embodiment of the present disclosure is shown.

[0023] Figure 8 This schematic diagram illustrates the replacement of the first adapter according to an embodiment of the present disclosure;

[0024] Figure 9 A flowchart illustrating an image processing method according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 10 A schematic block diagram of a model training apparatus according to an embodiment of the present disclosure is shown.

[0026] Figure 11 A schematic block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0027] Figure 12 A block diagram schematically illustrates an electronic device suitable for implementing a model training method or an image processing method according to embodiments of the present disclosure. Detailed Implementation

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and methods are not described in detail in order to avoid obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the stated features, steps, operations and / or components are present, but do not preclude the presence or addition of one or more other features, steps, operations, components, and / or groups thereof.

[0030] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the specification, and should not be interpreted in an idealized or overly formal manner.

[0031] In the case of using expressions similar to "at least one of A, B, and C, etc.", in general, it should be interpreted as including one or more of the corresponding items (for example, "a system having at least one of A, B, and C" should be interpreted to include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0032] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (such as including but not limited to user personal information) involved in the technical solutions comply with the relevant legal regulations, necessary security measures are taken, and do not violate public order and good customs.

[0033] According to the embodiments of the present disclosure, in the training process of the model, it is usually completed through a supervised algorithm of a specific processing task, but since the training samples are obtained by labeling the data, a lot of human resources are consumed, therefore, unsupervised training can be used to pre-train the model, and then globally adjust the model parameters of the pre-trained model to make the pre-trained model can process new processing tasks.

[0034] However, with the increase of the amount of data in the model and the gradual increase of the volume of the model, globally adjusting the model parameters of the pre-trained model not only consumes more resources, but also makes the model migration complex. For some special processing tasks, such as remote sensing image processing tasks, there is a problem that the training effect of the model is difficult to meet the demand due to the difficulty of obtaining sample images for training.

[0035] With the continuous increase of computing resources, the pre-trained model with the main architecture of the transformer has entered a new era. With more and more large models being proposed, solutions are provided for different tasks, but many new problems are also brought. Compared with general deep models, these large models trained by massive data contain more parameters.

[0036] Therefore, in the case of retraining the model for different processing tasks, storing the parameters of such large models will occupy more storage space, and training such large models will consume more resources and time. Based on this, the inventors found that by using a lightweight and highly scalable adapter, the trained model can meet the demand with only a small amount of training and small storage space.

[0037] In the process of implementing the present inventive concept, the inventors found that for some processing tasks, such as remote sensing image processing tasks, the number of sample images is small due to the difficulty of obtaining sample images, so even if an adapter is used, it is difficult to make the trained model meet the demand.

[0038] Based on this, the inventors found that by using the prompt learning method, the new processing task can be adjusted into the form of a similar pre-training task, and by using prompt learning, the model parameters in the pre-trained model can be fully utilized, thereby helping to train the pre-trained model.

[0039] Therefore, the inventors found that by combining prompt learning and adapters, the number of sample images required for training can be reduced, and the trained model can meet the demand.

[0040] Specifically, the embodiment of the present disclosure provides a model training method, comprising: in response to a first visual processing task for training a pre-trained model, obtaining a first visual processing task template, wherein the pre-trained model comprises a backbone network and a first adapter corresponding to the first visual processing task, and model parameters of the backbone network are obtained through pre-training; inputting the first visual processing task template and a sample image into the pre-trained model, processing the sample image and the first visual processing task template by using the first adapter to obtain a first visual processing result; training the pre-trained model by using the first visual processing result and a first visual processing label corresponding to the sample image to obtain a first visual processing task model, wherein the model parameters of the backbone network remain unchanged in the training process of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task.

[0041] Figure 1 An application scenario diagram of the model training method or the image processing method according to the embodiment of the present disclosure is schematically shown.

[0042] As shown in Figure 1 The application scenario 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] A user can use at least one of the first terminal device 101, the second terminal device 102 and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102 and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0044] The first terminal device 101, the second terminal device 102 and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers and desktop computers, etc.

[0045] The server 105 can be a server that provides various services, such as a background management server that provides support for a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing results (such as a web page, information, or data obtained or generated according to a user request, or the like) to the terminal device.

[0046] It should be noted that the model training method or the image processing method provided in the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the model training apparatus or the image processing apparatus provided in the embodiments of the present disclosure can generally be arranged in the server 105. The model training method or the image processing method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the model training apparatus or the image processing apparatus provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above scenario is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0048] The model training method of the embodiments of the present disclosure will be described in detail below based on the scenario described above. Figure 1 Figures 2-9 The model training method of the embodiments of the present disclosure will be described in detail below based on the scenario described above.

[0049] Figure 2 A flowchart of the model training method according to the embodiments of the present disclosure is schematically shown.

[0050] As shown in Figure 2 , the model training of this embodiment includes operations S210-S230.

[0051] In operation S210, in response to a first visual processing task for training a pre-trained model, a first visual processing task template is obtained, wherein the pre-trained model includes a backbone network and a first adapter corresponding to the first visual processing task, and the model parameters of the backbone network are obtained by pre-training.

[0052] ​According to an embodiment of the present disclosure, the pre-trained model can be a model obtained by pre-training in advance. For example, the pre-trained model can be a CNN (Convolutional Neural Network) pre-trained, but is not limited thereto. For example, the pre-trained model can be trained by a self-supervised algorithm, but is not limited thereto.

[0053] The pre-trained model can be used to process other processing tasks, for example, can be used to process an image target classification task, but is not limited thereto. For example, the image target classification task can be classification according to the shape of a target object, without specifically determining to which category the target object belongs.

[0054] According to an embodiment of the present disclosure, the first visual processing task can be a task different from the task processed by the pre-trained model, for example, can be an image target detection task, but is not limited thereto. The image target detection task can be detecting category information, size information, etc. of a target object in an image, but is not limited thereto, as long as it is different from the visual processing task corresponding to the pre-trained model.

[0055] According to an embodiment of the present disclosure, the first adapter can be untrained, can be partially trained corresponding to the first visual processing task, or can have prior knowledge corresponding to the first visual processing task. For example, in the case that the first adapter is partially trained corresponding to the first visual processing task, or has prior knowledge corresponding to the first visual processing task, the training speed of training the first adapter can be improved.

[0056] The first adapter can be arranged in the pre-trained model, for example, can be arranged between at least two network blocks included in the backbone network; the first adapter can also be arranged at the output end of the backbone network in the pre-trained model; the first adapter can also be arranged at the input end of the backbone network in the pre-trained model. For example, multiple first adapters can be arranged in the pre-trained model, and the arrangement positions are as described above, which will not be repeated here.

[0057] In operation S220, the first visual processing task template and the sample image are input into the pre-trained model, the sample image and the first visual processing task template are processed by the first adapter, and a first visual processing result is obtained.

[0058] According to an embodiment of the present disclosure, the sample image can correspond to the first visual processing task, for example, in the case that the first visual processing task is a remote sensing image target detection task, the sample image can correspond to a remote sensing image.

[0059] According to an embodiment of the present disclosure, the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, and the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task. The first visual processing result can be a result output by the pre-trained model in a training process, and the first visual processing result can correspond to the first visual processing task.

[0060] For example, in the case of the first visual processing task being an image object detection task, the first visual processing result can correspond to an image object detection result. The detection result can include class information, size information, and position information of the target object, but is not limited thereto.

[0061] The first visual processing task template can be a prompt learning template. Through the prompt learning method, the training speed of the pre-trained model can be improved, and in the case of a small number of sample images, the training of the model can also meet the demand. Therefore, the first visual processing result obtained by the first visual processing task template can correspond to the answer space of the prompt learning method.

[0062] According to an embodiment of the present disclosure, the template image can include a target object, and the target object in the template image can be detected to obtain a detection result. For example, the template image can include a remote sensing image. The label information can be generated by labeling the target object in the template image based on the detection result.

[0063] For example, the target object in the template image can be subjected to a labeling operation, and other objects can not be subjected to the labeling operation. The labeling operation can be labeling the position information of the target object in the template image, or labeling the size information of the target object in the template image, or labeling the class information of the target object in the template image, but is not limited thereto.

[0064] Therefore, in the case of training the model, the pre-trained model can process the sample image based on the label information and the target object in the template image corresponding to the label information to output the first visual processing result, instead of outputting the first visual processing result based on all objects in the sample image. Based on this, the guidance of the first visual processing result by the label information can be realized, and the demand for the number of sample images in the model training process can be reduced, and a satisfactory training effect can also be achieved using only a small number of sample images.

[0065] In operation S230, the pre-trained model is trained using the first visual processing result and the first visual processing label corresponding to the sample image to obtain the first visual processing task model, wherein the model parameters of the backbone network remain unchanged in the training process of the pre-trained model.

[0066] According to an embodiment of the present disclosure, the model parameters of the backbone network remain unchanged in the training process of the pre-training model, so that only the parameter information of the first adapter can be trained in the process of training the pre-training model.

[0067] According to an embodiment of the present disclosure, the first visual processing label corresponds to the sample image. For example, when the first visual processing task is a remote sensing image target detection task, the sample image can correspond to a remote sensing image, and the first visual processing label corresponds to the real information of the target object in the sample image, for example, the real information can include the real category of the target object and the real size of the target object, etc.

[0068] According to an embodiment of the present disclosure, the pre-training model is trained by using the first visual processing result and the first visual processing label corresponding to the sample image. For example, the loss information is determined by determining the first visual processing combination and the first visual processing label, and then the parameter information of the first adapter in the pre-training model is adjusted according to the loss information to obtain the first visual processing task model.

[0069] According to an embodiment of the present disclosure, since the pre-training model is trained to obtain the first visual processing task model, rather than training the untrained model to obtain the first visual processing task model, the training effect that meets the demand can be achieved in the case of training the pre-training model using fewer sample images compared with the untrained model, and resources can be saved and the training efficiency can be improved.

[0070] In addition, by setting the first adapter, the fitting ability of the pre-training model to the first visual processing task can be enhanced, the performance of the first visual task processing model obtained by training can be improved, and the first visual task processing model obtained by training can meet the demand.

[0071] Figure 3 An illustrative diagram of obtaining a first visual processing task model according to an embodiment of the present disclosure is shown.

[0072] As Figure 3 shown, the first visual processing task template 310 and the sample image 320 are input into the pre-training model 330 including the first adapter, and the first visual processing result 340 is output. Then, the pre-training model 330 including the first adapter is trained according to the first visual processing result 340 and the first visual processing label 350 to obtain the first visual processing task model 370.

[0073] According to an embodiment of the present disclosure, by keeping the model parameters of the backbone network unchanged in the training process of the pre-training model and adjusting only the parameters of the first adapter, the fitting capability of the pre-training model for the first visual processing task is enhanced through the adapter, global adjustment of the parameters of the pre-training model is avoided, the memory required for storing the model parameters is reduced, the resources of the pre-training model are fully utilized, the sample images required in the training process are saved, and the first visual processing result is guided through the first visual processing task template. Thus, by combining the first adapter with the first visual processing task template, the generalization capability of the pre-training model in the training process is enhanced, the training effect meeting the requirements can be achieved using only a small number of sample images, a first visual processing task model meeting the requirements is obtained, the resources of the sample images are saved, and the training efficiency is improved.

[0074] According to an embodiment of the present disclosure, before the first visual processing task template is obtained in response to the first visual processing task for training the pre-training model, the model training method further includes: obtaining a template image based on the first visual processing task; and obtaining the first visual processing task template and a first visual processing label based on the template image.

[0075] According to an embodiment of the present disclosure, the template image can be obtained from another electronic device or pre-stored in the electronic device, but is not limited thereto.

[0076] In a case where the first visual processing task is an image object detection task, the template image can be a corresponding detected image.

[0077] The label information can be obtained based on the detection result of the detected image and the detected image, and the first visual processing task template can be obtained based on the label information and the image.

[0078] According to an embodiment of the present disclosure, in a case where the first visual processing task is an image object detection task, the first visual processing label can further include class information and size information of an object in the detected template image.

[0079] According to an embodiment of the present disclosure, by obtaining the template image based on the first visual processing task and obtaining the first visual processing task template and the first visual processing label based on the template image, the first visual processing task template and the first visual processing label can be applied to the training process of the pre-training model, and thus the model training efficiency can be improved and the training resources can be saved.

[0080] According to an embodiment of the present disclosure, the pre-training model is trained by using the first visual processing result and the first visual processing label corresponding to the sample image to obtain a first visual processing task model, including: obtaining loss information according to the first visual processing result and the first visual processing label; and performing reverse gradient optimization on parameter information of the first adapter based on the loss information until the loss information meets a preset condition, to obtain the first visual processing task model.

[0081] According to an embodiment of the present disclosure, the loss information can be used to adjust the parameter information of the first adapter. The loss information can include a loss value obtained by processing the first visual processing result and the first visual processing label by using a loss function. For example, the loss value can be obtained by processing the first visual processing result and the first visual processing label by using a cross-entropy loss function, but is not limited thereto, and the loss value can also be obtained by using a loss function such as a cosine similarity loss function, and the present disclosure will not be repeated here.

[0082] According to an embodiment of the present disclosure, the preset condition can be a preset loss value threshold, and the parameter information of the first adapter can be adjusted to make the loss value less than or equal to the preset loss value threshold. In the case where the loss value is less than or equal to the preset loss value threshold, the first visual processing task model can be obtained according to the corresponding parameter information. The preset loss value threshold can be, for example, 10%, 15%, etc., and the present disclosure is not limited thereto.

[0083] Figure 4 A schematic diagram for obtaining loss information according to an embodiment of the present disclosure is schematically shown.

[0084] As shown in Figure 4 , the sample image 410 and the first visual processing task template 420 can be input into the pre-training model 430, and the first visual processing result 440 can be output. Then, the loss information 460 can be obtained according to the first visual processing result 440 and the first visual processing label 450. The parameters of the first adapter in the pre-training model 430 can be adjusted according to the loss information 460.

[0085] According to an embodiment of the present disclosure, the loss information is obtained according to the first visual processing label and the first visual processing result guided by the first visual processing template, and then the parameter information of the first adapter is optimized by reverse gradient based on the loss information until the loss information meets the preset condition. Therefore, the training efficiency can be improved, the sample images can be saved, and the first visual processing task model meeting the requirements can be obtained by using fewer sample images.

[0086] According to an embodiment of the present disclosure, the backbone network of the pre-trained model comprises a target visual processing module and at least two target network blocks, the at least two target network blocks comprising a first target network block and a second target network block; a first adapter is arranged between the at least two target network blocks; the first visual processing task template and the sample image are input into the pre-trained model, and the sample image and the first visual processing task template are processed by using the first adapter to obtain a first visual processing result, comprising: inputting the sample image and the first visual processing task template into the first target network block to output a first feature map and a first template feature map; inputting the first feature map and the first template feature map into the first adapter to output a second feature map and a second template feature map; inputting the second feature map and the second template feature map into the second target network block to output a third feature map and a third template feature map; and inputting the third feature map and the third template feature map into the target visual processing module to output the first visual processing result.

[0087] According to an embodiment of the present disclosure, the target network block can perform convolution processing on the input image, but is not limited thereto; the first adapter can also perform convolution processing on the input image, but is not limited thereto. The processing manner of the target network block and the first adapter on the input image can correspond to the first visual processing task.

[0088] According to an embodiment of the present disclosure, the processing manner of the target visual processing module on the input image can also correspond to the first visual processing task. For example, in the case where the first visual processing task is an image target detection task, the target visual processing module can be correspondingly used for detecting the input image.

[0089] According to an embodiment of the present disclosure, a plurality of first adapters can be arranged between the at least two target network blocks, which will not be described herein again.

[0090] Figure 5 A schematic diagram of a first visual processing result acquisition method according to an embodiment of the present disclosure is schematically shown.

[0091] As shown in Figure 5 , the sample image 510 and the first visual processing task template 520 can be input into the first target network block 530 to output a first feature map and a first template feature map; the first feature map and the first template feature map are input into the first adapter 540 to output a second feature map and a second template feature map; the second feature map and the second template feature map are input into the second target network block 550 to output a third feature map and a third template feature map; and the third feature map and the third template feature map are input into the target visual processing module 560 to output the first visual processing result 570.

[0092] According to an embodiment of the present disclosure, by inputting the sample image and the first visual processing task template into the first target network block, outputting the first feature map and the first template feature map, inputting the first feature map and the first template feature map into the first adapter, outputting the second feature map and the second template feature map, inputting the second feature map and the second template feature map into the second target network block, obtaining the third feature map and the third template feature map, and inputting the third feature map and the third template feature map into the target visual processing module, the first visual processing result is obtained. In this way, the first visual processing result is guided by the first visual processing task template, and the pre-trained model including the first adapter can be trained using the first visual processing result and the first visual processing label, thereby improving the model training efficiency and saving the sample images required for training the model.

[0093] According to an embodiment of the present disclosure, before the sample image is input into the pre-trained model, the sample image can be sliced first, and then the sliced image is input into the pre-trained model, so as to reduce the existing occupation in the training model process.

[0094] Figure 6 A schematic diagram of an output feature map set according to an embodiment of the present disclosure is schematically shown.

[0095] As shown in Figure 6 , the complete sample image can be sliced into sliced images I1…I m , and then the sliced images I1…I m are input into the backbone network of the pre-trained model. The backbone network can include an encoder (Embed), so that a corresponding feature map set E0 corresponding to the sliced images I1…I m can be output.

[0096] Figure 7 A schematic diagram of a training process according to an embodiment of the present disclosure is schematically shown.

[0097] As shown in Figure 7 , the feature map set E0 and the first visual processing task template P0 can be input into the transformer encoder layers L1…L N included in the network block of the backbone network, so that the third feature map and the third template feature map can be obtained, and the third feature map and the third template feature map can be input into the detector to obtain the first visual processing result.

[0098] According to an embodiment of the present disclosure, the above model training method further includes: using a trained second adapter corresponding to a second visual processing task to replace the first adapter in the first visual processing task model to obtain a second visual processing task model, wherein the second visual processing task model is used for processing a second visual processing task of an image, and the trained second adapter is obtained by training the pre-trained model based on the second visual processing task.

[0099] According to embodiments of this disclosure, the training method for the second adapter may include: setting the second adapter on a pre-trained model that does not include the first adapter; inputting a second visual processing task template and sample images corresponding to the second visual processing task into the pre-trained model including the second adapter to obtain a second visual processing result; and training the pre-trained model based on the second visual processing result and the second visual processing labels corresponding to the sample images to obtain a second visual processing task model including the second adapter.

[0100] According to embodiments of this disclosure, the position of the second adapter in the second vision processing task model can be the same as the position of the first adapter in the first vision processing task model.

[0101] According to embodiments of this disclosure, the first adapter in the first vision processing task model can be removed, and the second adapter can be placed in the same position. This allows the first adapter and the second adapter to be swapped, transforming the first vision processing task model into a second vision processing task model so that the model can be used to process the second vision processing task.

[0102] Figure 8 A schematic diagram illustrating the replacement of the first adapter according to an embodiment of the present disclosure is shown.

[0103] like Figure 8 As shown, the first vision processing task model 810 may include a first target network block 811, a first adapter 812, and a second target network block 813, and the second vision processing task model 820 may include a first target network block 811, a second adapter 821, and a second target network block 813.

[0104] According to embodiments of this disclosure, since only the parameters of the first adapter are adjusted during the training of the pre-trained model using the model training method of this disclosure, the model parameters of the first visual processing task model can be stored simply by saving the parameters of the trained first adapter and the model parameters of the backbone network of the pre-trained model. Similarly, a second visual processing task model can be trained using a similar model training method. This second visual processing task model may include a trained second adapter, and the model training method can refer to the model training method of this disclosure, which will not be described in detail here. Therefore, switching between the first and second visual processing task models can be achieved by swapping the trained first adapter and the trained second adapter, avoiding the need to directly save both the model parameters of the first and second visual processing task models. The same effect can be achieved by saving only the parameters of the trained first adapter and the trained second adapter, thereby reducing the memory usage of storing model parameters.

[0105] According to an embodiment of the present disclosure, by using the trained second adapter to replace the first adapter, the conversion between the first visual processing task model and the second visual processing task model can be realized without retraining, thereby saving training resources.

[0106] According to an embodiment of the present disclosure, before the first visual processing task template and the sample image are input into the pre-trained model, the sample image and the first visual processing task template are processed by the first adapter to obtain the first visual processing result, the model training method further includes: in response to receiving a sample acquisition instruction from the electronic device, determining the sample image from the sample database; and calling a sample transmission interface to acquire the sample image from the sample database.

[0107] According to an embodiment of the present disclosure, the sample label generation instruction can be from other electronic devices or pre-stored in the instruction database by the electronic device. The sample label generation instruction can include identification information of the sample to be acquired, so that the sample image required can be efficiently determined from the sample database storing a large number of sample images.

[0108] According to an embodiment of the present disclosure, the sample transmission interface can be a data interface pre-set on the electronic device, for example, a serial communication interface, etc.

[0109] According to an embodiment of the present disclosure, the sample database can be pre-stored on the electronic device, or stored on other electronic devices. Through the sample transmission interface, the sample image can be acquired from other electronic devices.

[0110] According to an embodiment of the present disclosure, by calling the sample transmission interface from the sample database according to the sample determination instruction by the electronic device, the sample image can be efficiently acquired, and the acquisition efficiency can meet the demand.

[0111] Figure 9 A flowchart of an image processing method according to an embodiment of the present disclosure is schematically shown.

[0112] As shown in Figure 9 the image processing method of this embodiment includes operation S910 to operation S930.

[0113] In operation S910, a to-be-processed image is acquired.

[0114] In operation S920, a first visual processing task model is obtained according to the model training method of the present disclosure.

[0115] In operation S930, the to-be-processed image is input into the first visual processing task model, and a target visual processing result is output.

[0116] According to an embodiment of the present disclosure, the image to be processed can be an image of the same type as the sample image, for example, in the case where the sample image is a remote sensing image, the image to be processed can also correspond to a remote sensing image.

[0117] According to an embodiment of the present disclosure, by keeping the model parameters of the backbone network unchanged in the training process of the pre-trained model and adjusting only the parameters of the first adapter, the fitting capability of the pre-trained model to the first visual processing task is enhanced through the adapter, global adjustment of the parameters of the pre-trained model is avoided, the memory required for storing the model parameters is reduced, the resources of the pre-trained model are fully utilized, the sample images required in the training process are saved, and the first visual processing result is guided through the first visual processing task template. Thus, by combining the first adapter with the first visual processing task template, the generalization capability of the pre-trained model in the training process is enhanced, the training effect meeting the requirements can be achieved using only a small amount of sample images, a first visual processing task model meeting the requirements is obtained, the resources of the sample images are saved, and the training efficiency is improved.

[0118] Based on the above model training method, the present disclosure further provides a model training device. The following will be described in combination with Figure 10 The device will be described in detail.

[0119] Figure 10 The structure block diagram of the model training device according to an embodiment of the present disclosure is schematically shown.

[0120] As Figure 10 shown, the model training device 1000 of this embodiment includes a first acquisition module 1010, a first input module 1020, and a first training module 1030.

[0121] The first acquisition module 1010 is configured to acquire a first visual processing task template in response to a first visual processing task for training a pre-trained model, wherein the pre-trained model includes a backbone network and a first adapter corresponding to the first visual processing task, and the model parameters of the backbone network are obtained through pre-training. In an embodiment, the first acquisition module 1010 can be configured to perform the operation S210 described in the foregoing, and details are not described herein again.

[0122] The first input module 1020 is configured to input the first visual processing task template and a sample image into the pre-trained model, process the sample image and the first visual processing task template by using the first adapter, and obtain a first visual processing result. In an embodiment, the first input module 1020 can be configured to perform the operation S220 described in the foregoing, and details are not described herein again.

[0123] The first training module 1030 is configured to train the pre-training model by using the first visual processing result and the first visual processing label corresponding to the sample image, to obtain a first visual processing task model, wherein the model parameters of the backbone network remain unchanged in the training process of the pre-training model; wherein the first visual processing task template includes a template image and label information about the template image, the template image is used to construct the first visual processing task template, the label information is used to guide the pre-training model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task. In an embodiment, the first training module 1030 can be configured to perform the operation S230 described above, and details are not described herein again.

[0124] It should be noted that the model training apparatus described above corresponds to the model training method described above, and the model training apparatus can include modules, units, sub-units, etc. for implementing all functions of the model training method involved in the flowchart. For the sake of brevity of description, details are not described herein again, and specific descriptions can be referred to the description of the model training method described above.

[0125] According to embodiments of the present disclosure, any of the first acquisition module 1010, the first input module 1020, and the first training module 1030 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of the modules can be combined with at least part of the functions of other modules, and implemented in one module. According to embodiments of the present disclosure, at least one of the first acquisition module 1010, the first input module 1020, and the first training module 1030 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged with a circuit, or implemented in any one of software, hardware, and firmware or in a proper combination of any of them. Alternatively, at least one of the first acquisition module 1010, the first input module 1020, and the first training module 1030 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0126] Based on the above processing method, the present disclosure further provides an image processing apparatus. The following will be described in combination with Figure 11 The apparatus will be described in detail.

[0127] Figure 11 The structure block diagram of the image processing apparatus according to embodiments of the present disclosure is schematically shown.

[0128] As Figure 11As shown, the image processing apparatus 1100 of this embodiment includes a second acquisition module 1110, a second training module 1120, and a second input module 1130.

[0129] The second acquisition module 1110 is configured to acquire an image to be processed. In an embodiment, the second acquisition module 1110 can be configured to perform operation S910 described above, and details are not repeated here.

[0130] The second training module 1120 is configured to obtain a first visual processing task model according to the model training method of the present disclosure. In an embodiment, the second training module 1120 can be configured to perform operation S920 described above, and details are not repeated here.

[0131] The second input module 1130 is configured to input the image to be processed into the first visual processing task model, and output a target visual processing result. In an embodiment, the second input module 1130 can be configured to perform operation S930 described above, and details are not repeated here.

[0132] It should be noted that the image processing apparatus described above corresponds to the image processing method described above, and the image processing apparatus can include modules, units, sub-units, etc. for implementing all functions of the image processing method involved in the above flowchart. For the sake of brevity, details are not repeated here, and specific descriptions can be referred to the description of the image processing method above.

[0133] According to embodiments of the present disclosure, any of the second acquisition module 1110, the second training module 1120, and the second input module 1130 can be combined in one module, or any of them can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules, and implemented in one module. According to embodiments of the present disclosure, at least one of the second acquisition module 1110, the second training module 1120, and the second input module 1130 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware that can be integrated or packaged, or any one of software, hardware, and firmware or any appropriate combination of several of them. Alternatively, at least one of the second acquisition module 1110, the second training module 1120, and the second input module 1130 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0134] Figure 12A block diagram of an electronic device suitable for implementing a model training method or an image processing method according to an embodiment of the present disclosure is schematically shown.

[0135] As shown in Figure 12 The electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage section 1208. The processor 1201 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), and / or the like. The processor 1201 can also include an on-board memory for cache use. The processor 1201 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to embodiments of the present disclosure.

[0136] In the RAM 1203, various programs and data required for the operation of the electronic device 1200 are stored. The processor 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. The processor 1201 performs various operations of the method processes according to embodiments of the present disclosure by executing programs in the ROM 1202 and / or the RAM 1203. Note that the programs can also be stored in one or more memories other than the ROM 1202 and the RAM 1203. The processor 1201 can also perform various operations of the method processes according to embodiments of the present disclosure by executing programs stored in the one or more memories.

[0137] According to an embodiment of the present disclosure, the electronic device 1200 can further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 can further include one or more of the following components connected to the input / output (I / O) interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage section 1208 including a hard disk, and the like; and a communication section 1209 including a network interface card such as a LAN card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output (I / O) interface 1205 as necessary. A removable media 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 1210 as necessary, so that a computer program read therefrom is installed into the storage section 1208 as necessary.

[0138] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or exist independently without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which when executed, implement the method according to the embodiments of the present disclosure.

[0139] According to embodiments of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer readable storage medium can include one or more of the above-described ROM 1202 and / or RAM 1203 and / or one or more memories other than the ROM 1202 and the RAM 1203.

[0140] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the model training method or the image processing method provided by the embodiments of the present disclosure.

[0141] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1201. According to embodiments of the present disclosure, the above-described systems, apparatuses, modules, units, and the like can be implemented by computer program modules.

[0142] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, and the like. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal in a network medium, and be downloaded and installed through the communication part 1209 and / or installed from the detachable medium 1211. The program codes contained in the computer program can be transmitted in any appropriate network medium, including but not limited to wireless, wired, or any appropriate combination thereof.

[0143] In such embodiments, the computer program can be downloaded and installed from the network through the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to the embodiments of the present disclosure, the system, device, apparatus, module, unit, and the like described above can be implemented by computer program modules.

[0144] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected through the Internet by using an Internet service provider).

[0145] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0146] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All these combinations and / or integrations fall within the scope of the present disclosure.

[0147] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A model training method, comprising: obtaining a first visual processing task template in response to a first visual processing task for training a pre-trained model, wherein the pre-trained model comprises a backbone network and a first adapter corresponding to the first visual processing task, and model parameters of the backbone network are obtained by pre-training; inputting the first visual processing task template and a sample image into the pre-trained model, processing the sample image and the first visual processing task template by using the first adapter, and obtaining a first visual processing result; training the pre-trained model by using the first visual processing result and a first visual processing label corresponding to the sample image, and obtaining a first visual processing task model, wherein model parameters of the backbone network remain unchanged in a training process of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, and the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task; wherein the training of the pre-trained model by using the first visual processing result and the first visual processing label corresponding to the sample image to obtain the first visual processing task model comprises: obtaining loss information according to the first visual processing result and the first visual processing label; performing reverse gradient optimization on parameter information of the first adapter based on the loss information until the loss information meets a preset condition, and obtaining the first visual processing task model; wherein the backbone network of the pre-trained model comprises a target visual processing module and at least two target network blocks, the at least two target network blocks comprise a first target network block and a second target network block, the first adapter is arranged between the at least two target network blocks, and the inputting of the first visual processing task template and the sample image into the pre-trained model, the processing of the sample image and the first visual processing task template by using the first adapter, and the obtaining of the first visual processing result comprise: inputting the sample image and the first visual processing task template into the first target network block to output a first feature map and a first template feature map; inputting the first feature map and the first template feature map into the first adapter to output a second feature map and a second template feature map; inputting the second feature map and the second template feature map into the second target network block to output a third feature map and a third template feature map; and inputting the third feature map and the third template feature map into the target visual processing module to output the first visual processing result.

2. The method of claim 1, wherein, Before the obtaining of the first visual processing task template in response to the first visual processing task for training the pre-trained model, the method further comprises: obtaining the template image based on the first visual processing task; obtaining the first visual processing task template and the first visual processing label based on the template image.

3. The method of claim 1, further comprising: replacing the first adapter in the first visual processing task model with a trained second adapter corresponding to a second visual processing task, to obtain a second visual processing task model, wherein the second visual processing task model is configured to process the second visual processing task of the image, and the trained second adapter is trained based on the second visual processing task.

4. The method of claim 1, wherein, before the inputting the first visual processing task template and the sample image into the pre-trained model and processing the sample image and the first visual processing task template using the first adapter to obtain the first visual processing result, the method further comprises: in response to receiving a sample acquisition instruction from the electronic device, determining the sample image from a sample database; calling a sample transmission interface to acquire the sample image from the sample database.

5. An image processing method, comprising: acquiring a to-be-processed image; obtaining the first visual processing task model according to any one of the methods of claims 1 to 4; inputting the to-be-processed image into the first visual processing task model to output a target visual processing result.

6. A model training apparatus for implementing the model training method of any one of claims 1 to 4, comprising: a first acquisition module configured to acquire a first visual processing task template in response to a first visual processing task for training a pre-trained model, wherein the pre-trained model comprises a backbone network and a first adapter corresponding to the first visual processing task, and model parameters of the backbone network are obtained by pre-training; a first input module configured to input the first visual processing task template and a sample image into the pre-trained model, and process the sample image and the first visual processing task template using the first adapter to obtain a first visual processing result; a first training module configured to train the pre-trained model using the first visual processing result and a first visual processing label corresponding to the sample image to obtain a first visual processing task model, wherein model parameters of the backbone network remain unchanged during the training of the pre-trained model; wherein the first visual processing task template comprises a template image and label information about the template image, the template image is used to construct the first visual processing task template, and the label information is used to guide the pre-trained model to output the first visual processing result, and the first visual processing task model is used to process the first visual processing task.

7. An electronic device, comprising: one or more processors; a storage apparatus configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 5.

Images