Weak supervision underlying visual color imaging method and device based on multi-prompt learning

Abstract

The invention provides a weak supervision underlying visual color imaging method and device based on multi-prompt learning. The method comprises the steps of obtaining six data sets, and generating a training set and weak supervision prompt information of similar tasks; constructing an end-to-end CLIP network based on a deep learning large model as a teacher network, and pre-training the teacher network by using the training set and the weak supervision prompt information to obtain a pre-trained teacher network; constructing a student network, learning multi-task common prompt information by using the training set, KL divergence and a mean square error loss function, and performing knowledge distillation on the student network according to a knowledge distillation function by using the multi-task common prompt information to obtain a distilled student network; and inputting a color-related task into the distilled student network, and outputting a target image of the corresponding task. In this way, only a single model structure is used for unifying bottom layer color tasks, only a small number of parameters need to be learned, the overhead of the calculation amount is reduced, and better color feature representation is obtained.

Description

Technical Field

[0001] The present invention generally relates to the fields of machine vision and artificial intelligence, and more specifically, to a weakly supervised underlying visual color imaging method and apparatus based on multi-cue learning. Background Art

[0002] Color is a crucial but inherently unstable visual attribute that is easily affected by changes in lighting color; the choice of color theme can also profoundly affect the viewer's perception of photos, artworks, and graphic designs. Therefore, color-related tasks have always been a fundamental research problem in the multimedia field and have broad application prospects in media editing and digital image processing. After changing the color of an image to obtain more accurate imaging results, the image can be widely used in many downstream visual tasks, such as object detection, instance segmentation, image retrieval, etc.

[0003] In recent years, with the development of GPU acceleration technology and the improvement of computer computing power, deep learning has made breakthrough progress in many fields, such as natural language processing and computer vision. In addition, with the introduction and rapid development of large models, large models can better improve the model's expressiveness, predictive ability, and generalization ability. Now large models have been widely used in many fields, but they have not received widespread attention in low-level visual tasks. Large models learn complex patterns and features by training massive amounts of data, and most low-level visual tasks face the problem of insufficient data. The prior knowledge of large models can make up for this problem well.

[0004] The key challenges facing the underlying color problem include: (1) the difficulty of annotating real images; (2) the lack of prior knowledge; and (3) the lack of understanding of contextual information. Summary of the invention

[0005] According to an embodiment of the present invention, a weakly supervised underlying visual color imaging scheme based on multi-cue learning is provided. This scheme uses only a single model structure to unify the underlying color tasks and only needs to learn a very small number of parameters, which greatly reduces the computational overhead and obtains better color feature representation.

[0006] In a first aspect of the present invention, a weakly supervised underlying visual color imaging method based on multi-cue learning is provided. The method comprises:

[0007] Obtain multi-illumination color constancy datasets, underwater image enhancement datasets, grayscale image colorization datasets, recoloring datasets, shadow removal datasets, and over / underexposure image restoration datasets to generate training sets; and generate weakly supervised prompt information for similar tasks;

[0008] An end-to-end CLIP network based on a deep learning large model is constructed as a teacher network, and the teacher network is pre-trained according to a mean square error loss function using a training set and weak supervision prompt information to obtain a pre-trained teacher network;

[0009] An end-to-end CLIP network structure based on knowledge distillation technology is constructed as a student network, and multi-task common prompt information is learned using a training set, KL divergence, and a mean square error loss function. The multi-task common prompt information is used to perform knowledge distillation on the student network according to a knowledge distillation function to obtain a distilled student network;

[0010] The color-related tasks are input into the distilled student network, and the target image of the corresponding task is output.

[0011] Furthermore, the method further comprises:

[0012] For the grayscale image coloring dataset and the recoloring dataset, an expected colored image is generated according to a target color palette, and the expected colored image is added to the training set as a true value.

[0013] Furthermore, the generating of weak supervision prompt information of similar tasks includes:

[0014] The multi-illumination color constancy task and underwater image enhancement task are regarded as similar tasks, and the standard 24-color color card image is used as the first weak supervision prompt information;

[0015] The grayscale image colorization task and the recolorization task are regarded as similar tasks, and the GT image palette sequence is used as the second weak supervision prompt information;

[0016] The shadow removal task and the over / under-exposure image restoration task are regarded as similar tasks, and edge information is used as the third weak supervisory cue information.

[0017] Furthermore, the teacher network has the same network structure as the student network, and the teacher network includes:

[0018] A feature extractor is used to obtain a training set, extract image features and prompt information features for image encoding, and output image encoding features;

[0019] The adapter is used to receive image coding features, adapt downstream tasks, and output the adapted image coding features;

[0020] The task output head is used to receive the adapted image encoding features and output the predicted image result with changed color.

[0021] Furthermore, the mean square error loss function is:

[0022] ;

[0023] in, For the The output image of each task is For the The target image for each task; For average operation; is the mean square error loss function.

[0024] Furthermore, the method of learning multi-task common prompt information by using a training set, a KL divergence and a mean square error loss function includes:

[0025] The CLIP image encoder is used to extract features of the first weakly supervised prompt information, the second weakly supervised prompt information, and the third weakly supervised prompt information, respectively, to obtain three weakly supervised prompt vectors, and the feature dimensions of the three weakly supervised prompt vectors are aligned, and the prompt vector is obtained by cascading them. ;

[0026] Initialize the parameters of the student network using the parameters of the teacher network, and set the random initialization prompt vector of the student network;

[0027] Use KL divergence to align randomly initialized hint vectors and , get multi-task common prompt information between tasks.

[0028] Furthermore, the knowledge distillation function is:

[0029] ;

[0030] in, is the KL divergence; A randomly initialized hint vector for the student network; is the prompt vector.

[0031] In a second aspect of the present invention, a weakly supervised underlying visual color imaging device based on multi-cue learning is provided. The device comprises:

[0032] The data acquisition module is used to acquire a multi-illumination color constancy dataset, an underwater image enhancement dataset, a grayscale image colorization dataset, a recoloring dataset, a shadow removal dataset, and an over / underexposure image restoration dataset to generate a training set; and to generate weakly supervised prompt information for similar tasks;

[0033] A network training module is used to construct an end-to-end CLIP network based on a deep learning large model as a teacher network, and pre-train the teacher network according to a mean square error loss function using a training set and weak supervision prompt information to obtain a pre-trained teacher network;

[0034] A knowledge distillation module is used to construct an end-to-end CLIP network structure based on knowledge distillation technology as a student network, learn multi-task common prompt information using a training set, KL divergence and mean square error loss function, and perform knowledge distillation on the student network according to a knowledge distillation function using the multi-task common prompt information to obtain a distilled student network;

[0035] The imaging module is used to input color-related tasks into the distilled student network and output a target image of the corresponding task.

[0036] In a third aspect of the present invention, an electronic device is provided. The electronic device has at least one processor; and a memory connected to the at least one processor in communication; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method of the first aspect of the present invention.

[0037] In a fourth aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the first aspect of the present invention.

[0038] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0040] Figure 1 A flowchart of a weakly supervised underlying visual color imaging method based on multi-cue learning according to an embodiment of the present invention is shown;

[0041] Figure 2 A block diagram of a weakly supervised underlying visual color imaging device based on multi-cue learning according to an embodiment of the present invention is shown;

[0042] Figure 3 A block diagram of an exemplary electronic device capable of implementing embodiments of the present invention is shown;

[0043] Among them, 300 is an electronic device, 301 is a computing unit, 302 is a ROM, 303 is a RAM, 304 is a bus, 305 is an I / O interface, 306 is an input unit, 307 is an output unit, 308 is a storage unit, and 309 is a communication unit. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0046] Figure 1 A flowchart of a weakly supervised underlying visual color imaging method based on multi-cue learning according to an embodiment of the present invention is shown.

[0047] The method includes:

[0048] S101. Obtain a multi-illumination color constancy dataset, an underwater image enhancement dataset, a grayscale image colorization dataset, a recoloring dataset, a shadow removal dataset, and an over / underexposure image restoration dataset to generate a training set; and generate weak supervision prompt information for similar tasks.

[0049] In this embodiment, the Large Scale Multi-Illuminant (LSMI) dataset is a dataset used to test algorithms that maintain color stability under different lighting conditions. This dataset contains images taken under different lighting conditions and the corresponding real color images under balanced lighting conditions.

[0050] The Underwater Image Enhancement Benchmark (UIEB) is a dataset from real underwater scenes suitable for underwater image enhancement research.

[0051] The Grayscale Image Colorization and Recolorization Dataset (ImageNet) is a large-scale image dataset that is a milestone in the field of computer vision and deep learning.

[0052] The Image Shadow Triplets Dataset (ISTD) is a challenging dataset containing triplets of shadow images, shadow-occluded images, and shadow-free images, supporting tasks such as shadow detection and shadow removal.

[0053] The low light (LOL) dataset consists of a set of low-light and normal-light image pairs, supporting the research and development of low-light image enhancement techniques and covering various indoor scenes.

[0054] In this embodiment, for the grayscale image coloring and recoloring datasets, preprocessing is required before generating training data. For example, 50,000 images are randomly selected from ImageNet as the datasets for grayscale image coloring and recoloring tasks, and preprocessing is performed to divide the training set and the test set into a ratio of 8:2.

[0055] The preprocessing includes: for a recoloring task, firstly generating a desired colored image according to a target palette, and then using the generated image as a ground truth (GT), and the target palette can be randomly obtained according to a user's selection.

[0056] In this embodiment, it is necessary to classify and divide tasks according to their similarity, including:

[0057] The datasets of the color constancy task and the underwater image enhancement task are obtained. The real illumination of the color constancy dataset is based on the assumption that there is only one light source in the scene, so the real result is not completely accurate; the dataset of the underwater image enhancement task does not have accurate enhanced image results. The above two tasks are regarded as similar tasks and classified into the same category.

[0058] Obtain datasets for grayscale image colorization and recolorization tasks, where the recolorization and grayscale image colorization tasks do not have practical and accurate pixel-level annotations, but only provide a small amount of color information. The above two tasks are regarded as similar tasks and classified into the same category.

[0059] Obtain datasets for shadow removal tasks and over / under exposure image restoration, where the shadow removal task does not give an accurate shadow area range; the over / under exposure image restoration dataset does not give an accurate exposure adjustment value. The above two tasks are considered similar tasks and classified into the same category.

[0060] In this embodiment, weak supervision prompt information of similar tasks is generated, including:

[0061] The main goal of the color constancy task and the underwater image enhancement task is to change the image illumination color. Therefore, the multi-illumination color constancy task and the underwater image enhancement task are taken as similar tasks, and the standard 24-color color card image ( ) as the first weakly supervised prompt information, wherein the first weakly supervised prompt information is used to assist in lighting color correction.

[0062] The main goal of the grayscale image colorization task and the recolorization task is to give the input image a new color. Therefore, the grayscale image colorization task and the recolorization task are regarded as similar tasks, and the GT image palette sequence ( ) as the second weakly supervised hint information, which is used to assist in learning the color distribution. The color palette is the main color clustering result of the target coloring image. In this task, 4-5 clustered target colors are generally obtained.

[0063] The main goal of the shadow removal task and the over / under exposure image restoration task is to unify the image color. Therefore, the shadow removal task and the over / under exposure image restoration task are regarded as similar tasks, and the edge information ( ) as the third weak supervision hint information, which is used to assist the input image in clear imaging. The edge information refers to the places where the pixel intensity in the image changes sharply, and these changes usually correspond to the boundaries of objects in the image. The edge is a key feature to distinguish different objects from the background.

[0064] S102. Construct an end-to-end CLIP network based on a deep learning large model as a teacher network, use a training set and weak supervision prompt information, and pre-train the teacher network according to a mean square error loss function to obtain a pre-trained teacher network.

[0065] In this embodiment, the teacher network includes:

[0066] The feature extractor is used to obtain a training set, extract image features and prompt information features for image encoding, and output image encoding features; the feature extractor adopts a CLIP image encoder.

[0067] The adapter is used to receive image coding features, adapt downstream tasks, and output the adapted image coding features;

[0068] The task output head is used to receive the adapted image encoding features and output the predicted image result with changed color.

[0069] In this embodiment, during pre-training, the parameters of the CLIP image encoder are kept fixed, and other parameters are trained.

[0070] In this embodiment, the mean square error loss function is used as the optimization objective function, and the mean square error loss function is:

[0071] ;

[0072] in, is the output image of the i-th task; is the target image of the i-th task; For average operation; is the mean square error loss function.

[0073] S103. Construct an end-to-end CLIP network structure based on knowledge distillation technology as a student network, use a training set, KL divergence and mean square error loss function to learn multi-task common prompt information, use the multi-task common prompt information, and train the student network according to the knowledge distillation function to obtain a trained student network.

[0074] In this embodiment, the network structure of the student network is the same as that of the teacher network, and will not be described in detail.

[0075] In this embodiment, the method of learning multi-task common prompt information by using a training set, a KL divergence and a mean square error loss function includes:

[0076] First, given an input image And the first weak supervision prompt information, the second weak supervision prompt information and the third weak supervision prompt information, where Represents a set of data; the input image is used as the input of the student network. The first weak supervision prompt information, the second weak supervision prompt information and the third weak supervision prompt information are respectively extracted using the CLIP image encoder to obtain three weak supervision prompt vectors, the three weak supervision prompt vectors are concatenated, and the feature dimensions of the three weak supervision prompt vectors are aligned using a convolutional layer, and the prompt vector is obtained by cascading them. ;

[0077] ;

[0078] in, is the prompt vector; is the first weakly supervised hint vector; is the second weakly supervised hint vector; is the third weakly supervised hint vector.

[0079] Secondly, the parameters of the student network are initialized using the parameters of the teacher network, and the random initialization prompt vector of the student network is set .

[0080] Finally, the KL divergence is used to align the randomly initialized hint vector and the prompt vector , and obtain the multi-task common prompt information between tasks. In this embodiment, it is the shared color features between the six tasks. The shared color features are used as soft prompts for the test tasks. Since the shared color features already contain the prompt information of the six tasks, the model can process multiple color tasks at the same time. The model performance is improved by sharing information between similar tasks.

[0081] During the knowledge distillation process, the CLIP image encoder is frozen and the parameters of the adapter, task output head, and hint vector are learnable.

[0082] In this embodiment, the knowledge distillation function is:

[0083] ;

[0084] in, is the KL divergence; A randomly initialized hint vector for the student network; is the concatenated hint vector.

[0085] S104: Input the color-related task into the distilled student network, and output the target image of the corresponding task.

[0086] In some embodiments, the distilled student model is used as a test model for testing.

[0087] Test image Input into CLIP image encoder to get features , with a randomly initialized hint vector for the student network Stitching; after the adapter optimizes the specific color task, the final output image color change result.

[0088] Different evaluation indicators are used for different color tasks:

[0089] Color constancy task—mean angular error;

[0090] Underwater image enhancement task - underwater image quality assessment;

[0091] Grayscale image colorization task—Fréchette distance;

[0092] Recoloring tasks — mostly use qualitative evaluation;

[0093] Shadow removal task—mean square error;

[0094] Over / under-exposed image restoration task—Peak signal-to-noise ratio (PSNR).

[0095] According to an embodiment of the present invention, the CLIP large model uses rich prior information about images and learnable prompt information to provide high-level auxiliary information for underlying color tasks and guide the model to generate specific images through prompts. At the same time, only one model is needed to meet different types of image color changes.

[0096] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0097] The above is an introduction to a method embodiment. The following is a further explanation of the scheme of the present invention through a device embodiment having the same inventive concept as the method in the aforementioned embodiment.

[0098] like Figure 2 As shown, the device 200 includes:

[0099] The data acquisition module 210 is used to acquire a multi-illumination color constancy dataset, an underwater image enhancement dataset, a grayscale image colorization dataset, a recoloring dataset, a shadow removal dataset, and an over / underexposure image restoration dataset to generate a training set; and to generate weak supervision prompt information for similar tasks;

[0100] A network training module 220 is used to construct an end-to-end CLIP network based on a deep learning large model as a teacher network, and pre-train the teacher network according to a mean square error loss function using a training set and weak supervision prompt information to obtain a pre-trained teacher network;

[0101] The knowledge distillation module 230 is used to construct an end-to-end CLIP network structure based on knowledge distillation technology as a student network, learn multi-task common prompt information using a training set, KL divergence and mean square error loss function, and perform knowledge distillation on the student network according to a knowledge distillation function using the multi-task common prompt information to obtain a distilled student network;

[0102] The imaging module 240 is used to input the color-related tasks into the distilled student network and output the target image of the corresponding task.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0104] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] According to an embodiment of the present invention, the present invention further provides an electronic device and a readable storage medium.

[0106] Figure 3 A schematic block diagram of an electronic device 300 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0107] The electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0108] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 306, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as methods S101 to S104. For example, in some embodiments, methods S101 to S104 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the methods S101 to S104 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute methods S101 - S104 in any other appropriate manner (eg, by means of firmware).

[0110] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A weakly supervised underlying visual color imaging method based on multi-cue learning, characterized in that: include: Obtain a multi-illumination color constancy dataset, an underwater image enhancement dataset, a grayscale image colorization dataset, a recoloring dataset, a shadow removal dataset, and an over / underexposure image restoration dataset to generate a training set; And generate weakly supervised prompt information for similar tasks; An end-to-end CLIP network based on a deep learning large model is constructed as a teacher network, and the teacher network is pre-trained according to a mean square error loss function using a training set and weak supervision prompt information to obtain a pre-trained teacher network; An end-to-end CLIP network structure based on knowledge distillation technology is constructed as a student network, and multi-task common prompt information is learned using a training set, KL divergence, and a mean square error loss function. The multi-task common prompt information is used to perform knowledge distillation on the student network according to a knowledge distillation function to obtain a distilled student network; The color-related tasks are input into the distilled student network, and the target image of the corresponding task is output.

2. The method according to claim 1, characterized in that Also includes: For the grayscale image coloring dataset and the recoloring dataset, an expected colored image is generated according to a target color palette, and the expected colored image is added to the training set as a true value.

3. The method according to claim 1, characterized in that The generating of weakly supervised prompt information of similar tasks includes: The multi-illumination color constancy task and underwater image enhancement task are regarded as similar tasks, and the standard 24-color color card image is used as the first weak supervision prompt information; The grayscale image colorization task and the recolorization task are regarded as similar tasks, and the GT image palette sequence is used as the second weak supervision prompt information; The shadow removal task and the over / under-exposure image restoration task are regarded as similar tasks, and edge information is used as the third weak supervisory cue information.

4. The method according to claim 1, characterized in that: The teacher network has the same network structure as the student network, and the teacher network includes: A feature extractor is used to obtain a training set, extract image features and prompt information features for image encoding, and output image encoding features; The adapter is used to receive image coding features, adapt downstream tasks, and output the adapted image coding features; The task output head is used to receive the adapted image encoding features and output the predicted image result with changed color.

5. The method according to claim 1, characterized in that The mean square error loss function is: ； in, For the The output image of each task is For the The target image for each task; For average operation; is the mean square error loss function.

6. The method according to claim 1, characterized in that The method of learning multi-task common prompt information by using a training set, a KL divergence and a mean square error loss function includes: The CLIP image encoder is used to extract features of the first weakly supervised prompt information, the second weakly supervised prompt information, and the third weakly supervised prompt information, respectively, to obtain three weakly supervised prompt vectors, and the feature dimensions of the three weakly supervised prompt vectors are aligned, and the prompt vector is obtained by cascading them. ; Initialize the parameters of the student network using the parameters of the teacher network, and set the random initialization prompt vector of the student network; Use KL divergence to align randomly initialized hint vectors and , get multi-task common prompt information between tasks.

7. The method according to claim 1, characterized in that The knowledge distillation function is: ； in, is the KL divergence; A randomly initialized hint vector for the student network; is the prompt vector.

8. A weakly supervised underlying visual color imaging device based on multi-cue learning, characterized in that: include: A data acquisition module is used to acquire a multi-illumination color constancy dataset, an underwater image enhancement dataset, a grayscale image colorization dataset, a recoloring dataset, a shadow removal dataset, and an over / underexposure image restoration dataset to generate a training set; And generate weakly supervised prompt information for similar tasks; A network training module is used to construct an end-to-end CLIP network based on a deep learning large model as a teacher network, and pre-train the teacher network according to a mean square error loss function using a training set and weak supervision prompt information to obtain a pre-trained teacher network; A knowledge distillation module is used to construct an end-to-end CLIP network structure based on knowledge distillation technology as a student network, learn multi-task common prompt information using a training set, KL divergence and mean square error loss function, and perform knowledge distillation on the student network according to a knowledge distillation function using the multi-task common prompt information to obtain a distilled student network; The imaging module is used to input color-related tasks into the distilled student network and output a target image of the corresponding task.

9. An electronic device comprising at least one processor; and a memory connected in communication with the at least one processor; characterized in that: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images