A training method of an image enhancement model, an image processing method, an image processing device, an electronic device, a computer readable storage medium and a computer program product

Abstract

The application provides a training method of an image enhancement model, an image processing method and device, an electronic device, a computer readable storage medium and a computer program product. The training method of the image enhancement model comprises: obtaining a sample image and a label image, the label image being a reference enhancement result of the sample image; enhancing the sample image through the image enhancement model to obtain an enhanced sample image; determining a first description text, the first description text being used for describing image content of the label image; extracting a label image semantic of the label image based on the first description text; extracting a first image feature of the enhanced sample image based on the label image semantic; determining a loss value of a target loss function based on the first image feature and the target loss function of the image enhancement model; and training the image enhancement model based on the loss value of the target loss function to obtain a trained image enhancement model. Through the application, the accuracy of the trained image enhancement model can be improved.

Description

Technical Field

[0001] This application relates to image processing technology in the field of artificial intelligence, and more particularly to a training method for an image enhancement model, an image processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Image enhancement can improve image quality during image processing. Generally, to enhance an image, a training model is first trained based on the sample image to be enhanced and the labeled enhanced image. The trained model is then used to enhance the image. However, training the model solely based on the labeled image negatively impacts the accuracy of the trained image enhancement model. Summary of the Invention

[0003] This application provides a training method, image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image enhancement model, which can improve the accuracy of the trained image enhancement model.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a training method for an image enhancement model, the training method comprising:

[0006] Obtain a sample image and a label image, wherein the label image is a reference enhancement result of the sample image;

[0007] The sample image is enhanced using an image enhancement model to obtain an enhanced sample image;

[0008] Determine a first descriptive text, which is used to describe the image content of the label image;

[0009] Extract the semantic meaning of the label image from the label image based on the first descriptive text;

[0010] Based on the semantics of the labeled image, the first image feature of the enhanced sample image is extracted;

[0011] Based on the first image features and the target loss function pre-set for the image enhancement model, the loss value of the target loss function is determined;

[0012] The image enhancement model is trained based on the loss value of the target loss function to obtain the trained image enhancement model.

[0013] This application provides an image processing method, the image processing method comprising:

[0014] In response to an image enhancement request, the image to be enhanced, which is requested to be enhanced, is obtained;

[0015] The image to be enhanced is enhanced by the trained image enhancement model to obtain the enhanced image. The trained image enhancement model is obtained by the training method of the image enhancement model provided in the embodiments of this application.

[0016] This application provides a training apparatus for an image enhancement model, the training apparatus comprising:

[0017] The model enhancement module is used to acquire sample images and label images, wherein the label image is a reference enhancement result of the sample image; and to enhance the sample image using an image enhancement model to obtain an enhanced sample image.

[0018] A text determination module is used to determine a first descriptive text, which is used to describe the image content of the label image;

[0019] The semantic extraction module is used to extract the semantic meaning of the label image based on the first descriptive text;

[0020] The loss determination module is used to extract the first image features of the enhanced sample image based on the semantics of the labeled image; and to determine the loss value of the target loss function based on the first image features and the target loss function pre-set for the image enhancement model.

[0021] The model training module is used to train the image enhancement model based on the loss value of the target loss function, so as to obtain the trained image enhancement model.

[0022] In this embodiment of the application, the target loss function includes a first adversarial loss function. The loss determination module is further configured to determine the loss value of the first adversarial loss function based on the first image features and the first adversarial loss function. The first adversarial loss function is used to characterize the mapping relationship between the first image features and the loss value of the first adversarial loss function in the adversarial learning mode of the image enhancement model. The loss value of the target loss function is determined based on the loss value of the first adversarial loss function.

[0023] In this embodiment of the application, the loss determination module is further configured to: determine a target query feature based on a first text feature and the semantics of the labeled image, wherein the first text feature is a feature of the first descriptive text and the target query feature is an information feature to be queried in the enhanced sample image; determine a first key feature and a first value feature of the enhanced sample image, wherein the first key feature is an information feature in the enhanced sample image to be compared with the target query feature and the first value feature is a content feature in the enhanced sample image corresponding to the first key feature; and perform attention processing on the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image.

[0024] In this embodiment of the application, the loss determination module is further configured to enhance the semantics of the labeled image to obtain the semantics of the target image; and to fuse the first text features and the semantics of the target image to obtain the target query features.

[0025] In this embodiment of the application, the loss determination module is further configured to determine the image semantic weights of the labeled image semantics, the image semantic weights being used to enhance the labeled image semantics; to perform convolution on the labeled image semantics to obtain image semantic features; and to fuse the image semantic weights and the image semantic features to obtain the target image semantics.

[0026] In this embodiment, the loss determination module is further configured to perform attention processing on the target query feature, the first key feature, and the first value feature to obtain intermediate image features of the enhanced sample image; determine the second key feature and the second value feature of the intermediate image features, wherein the second key feature is an information feature in the intermediate image features to be compared with the target query feature, and the second value feature is a content feature in the intermediate image features corresponding to the second key feature; and perform attention processing on the target query feature, the second key feature, and the second value feature to obtain the first image feature of the enhanced sample image.

[0027] In this embodiment of the application, the target loss function further includes a first loss function. The loss determination module is further configured to determine the loss value of the first loss function based on the enhanced sample image, the label image, and the first loss function. The loss value of the first loss function includes at least one of a model loss value and a perceptual loss value. The model loss value is calculated using the loss function corresponding to the image enhancement model, and the perceptual loss value is obtained by calculating the difference between the enhanced sample image and the label image in the feature space. Based on the loss value of the first loss function and the loss value of the first adversarial loss function, the loss value of the target loss function is determined.

[0028] In this embodiment of the application, the loss determination module is further configured to determine a first type label of the enhanced sample image, wherein the first type label is the true label of the enhanced sample image; and determine the loss value of the first adversarial loss function based on the first type label, the first image features, and the first adversarial loss function.

[0029] In this embodiment, the extraction of the first image feature of the enhanced sample image based on the semantics of the labeled image is implemented by a discriminator. The model training module is further configured to extract the second image feature of the labeled image based on the semantics of the labeled image using the discriminator; determine the loss value of a second adversarial loss function based on the second image feature and the first image feature, wherein the second adversarial loss function is used to characterize the mapping relationship between the second image feature and the first image feature and the loss value of the second adversarial loss function in the adversarial learning mode; and train the discriminator based on the loss value of the second adversarial loss function to obtain an intermediate discriminator.

[0030] In this embodiment of the application, the model training module is further configured to train the image enhancement model based on the loss value of the target loss function to obtain an intermediate image enhancement model; and to train the intermediate image enhancement model through the intermediate discriminator to obtain the trained image enhancement model.

[0031] In this embodiment of the application, the model training module is further configured to determine a second type label of the augmented sample image, wherein the second type label is a false label of the augmented sample image; determine a third type label of the label image, wherein the third type label is a true label of the label image; and calculate a loss value based on the second type label and the first image feature, and the third type label and the second image feature, to obtain the loss value of the second adversarial loss function.

[0032] In this embodiment of the application, the semantic extraction module is further configured to extract the first text features of the first descriptive text; extract the label image features of the label image; and fuse the first text features and the label image features to obtain the label image semantics.

[0033] In this embodiment of the application, the semantic extraction module is further configured to perform an affine transformation on the first text features to obtain scaling parameters and bias terms of the label image features; fuse the scaling parameters with the label image features to obtain a feature fusion result; and combine the label image features, the bias terms, and the feature fusion result into the label image semantics.

[0034] In this embodiment, the text determination module is further configured to determine a first initialization text of the label image and a second initialization text of an auxiliary sample image, wherein the auxiliary sample image is a sample image used to assist the label image in optimizing the first initialization text; determine a first similarity between the label image and the first initialization text, and a second similarity between the auxiliary sample image and the second initialization text; determine a loss value of a second loss function between the first initialization text and the label image based on the first similarity and the second similarity; and adjust the first initialization text based on the loss value of the second loss function to obtain the first descriptive text.

[0035] This application provides an image processing apparatus, the image processing apparatus comprising:

[0036] An image acquisition module is used to obtain the image to be enhanced as requested in the image enhancement request in response to the image enhancement request;

[0037] The model application module is used to enhance the image to be enhanced using a trained image enhancement model to obtain an enhanced image. The trained image enhancement model is obtained through the training method of the image enhancement model provided in the embodiments of this application.

[0038] In this embodiment of the application, the model training device further includes a result output module for displaying the enhanced image; or, determining the image classification result of the enhanced image and displaying the enhanced image and the image classification result.

[0039] This application provides an electronic device, the electronic device comprising:

[0040] Memory is used to store executable instructions or computer programs.

[0041] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method for the image enhancement model provided in the embodiments of this application, or implements the image processing method provided in the embodiments of this application.

[0042] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When executed by a processor, the computer-executable instructions or computer programs implement the training method for the image enhancement model provided in this application, or implement the image processing method provided in this application.

[0043] This application provides a computer program product, including computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, they implement the training method for the image enhancement model provided in this application, or the image processing method provided in this application.

[0044] The embodiments of this application have at least the following beneficial effects: after obtaining the enhanced sample image of the sample image through the image enhancement model, the label image semantics of the label image is extracted based on the first descriptive text corresponding to the label image of the sample image, and the first image features of the enhanced sample image are extracted based on the label image semantics. Based on the first image features and the target loss function pre-set for the image enhancement model, the loss value used to train the image enhancement model is determined, so as to realize the training of the image enhancement model. In the training process of the image enhancement model, the first image feature used to determine the loss value of the target loss function is extracted based on the semantics of the label image. This semantics is extracted from the label image based on the first descriptive text. The first descriptive text represents the textual information of the label image content, which is different from the label image itself. Therefore, extracting the semantics of the label image based on the first descriptive text is a process of determining the semantics of the label image through information from different dimensions. This improves the comprehensiveness of the label image semantics in the feature dimension, thus improving its accuracy in representing the enhancement result. Therefore, extracting the first image feature used to calculate the loss value of the target loss function based on the semantics of the label image improves the accuracy of the target loss function's loss value. Consequently, training the image enhancement model based on the loss value of this target loss function improves the accuracy of the trained image enhancement model. Attached Figure Description

[0045] Figure 1 This is an exemplary diagram illustrating model training by incorporating prior information;

[0046] Figure 2 This is another exemplary diagram illustrating model training by incorporating prior information;

[0047] Figure 3 This is a schematic diagram of the architecture of the image processing system provided in the embodiments of this application;

[0048] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0050] Figure 6AThis is a flowchart illustrating the training method of the image enhancement model provided in the embodiments of this application. Figure 1 ;

[0051] Figure 6B The embodiments provided in this application are related to Figure 6A The corresponding logic flow diagram;

[0052] Figure 7A This is a flowchart illustrating the training method of the image enhancement model provided in the embodiments of this application. Figure 2 ;

[0053] Figure 7B This is a schematic diagram of the logical flow for determining the first descriptive text provided in an embodiment of this application;

[0054] Figure 7C This is a schematic diagram of the logical flow for determining the semantics of a labeled image provided in an embodiment of this application;

[0055] Figure 7D This is a schematic diagram of the logic flow for obtaining the first image features provided in an embodiment of this application;

[0056] Figure 8A This is a schematic diagram of the process for obtaining the first image features provided in an embodiment of this application;

[0057] Figure 8B The embodiments provided in this application are related to Figure 8A The corresponding logic flow diagram;

[0058] Figure 9A This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0059] Figure 9B The embodiments provided in this application are related to Figure 9A The corresponding logic flow diagram;

[0060] Figure 10 This is a schematic diagram of an exemplary training image enhancement model provided in an embodiment of this application;

[0061] Figure 11 This is the process of obtaining prompt text for the label image provided in the embodiments of this application;

[0062] Figure 12 This is an exemplary text modulation diagram provided in an embodiment of this application;

[0063] Figure 13 This is a schematic diagram of an exemplary text modulation semantic awareness discriminator provided in an embodiment of this application;

[0064] Figure 14 This is a schematic diagram of an exemplary semantic guidance block provided in an embodiment of this application;

[0065] Figure 15 This is an exemplary model training flowchart provided in an embodiment of this application;

[0066] Figure 16 This is an exemplary model testing flowchart provided in an embodiment of this application;

[0067] Figure 17 This is an exemplary image enhancement diagram provided in an embodiment of this application;

[0068] Figure 18 This is another exemplary image enhancement diagram provided in the embodiments of this application. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0071] In the following description, the terms “first, second, third”, etc., are used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, third”, etc., may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0072] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0073] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0074] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0075] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0076] 1) An artificial neural network is a mathematical model that mimics the structure and function of biological neural networks. Exemplary structures of artificial neural networks in this application include Graph Convolutional Networks (GCNs, a type of neural network for processing graph-structured data), Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Neural State Machines (NSMs), and Phase-Functioned Neural Networks (PFNNs). The network structure implemented using artificial neural networks is called an artificial neural network model. In this application, the image enhancement model, the trained image enhancement model, the discriminator, the trained discriminator, and the image-text model are all artificial neural network models.

[0077] 2) Contrastive Language-Image Pre-Training (CLIP), also known as an image-text model, is used to learn the relationship between images and text to achieve efficient multimodal interaction and task processing. CLIP consists of two parts: an image encoder and a text encoder. The image encoder converts images into feature vectors, and the text encoder converts text into feature vectors. During pre-training, CLIP uses a matching metric (e.g., cosine similarity or other distance metrics) to evaluate the similarity between the feature vectors of the image and the feature vectors of the text, reducing the distance between matching image-text pairs in the feature space and increasing the distance between mismatched image-text pairs. CLIP can be applied to image classification, generation of descriptive text for images, text-based image search, image-based text search, and generation of image content corresponding to text. In the embodiments of this application, CLIP can be used to determine the descriptive text of an image.

[0078] The following describes the relevant techniques for image enhancement.

[0079] It should be noted that, in order to train an artificial neural network model for image enhancement, the image enhancement model can be trained first based on the sample image to be enhanced and the labeled enhanced image, and then the trained image enhancement model can be used to enhance the image. However, in the above model training process, relying solely on the labeled image affects the accuracy of the trained image enhancement model.

[0080] Furthermore, prior information can be incorporated into the training of artificial neural network models for image enhancement. In the field of artificial intelligence (AI), prior information refers to knowledge or assumptions known before reasoning, learning, or decision-making, typically derived from prior experience, domain knowledge, statistical data, or theoretical inferences. Prior information can provide domain-specific knowledge for AI processing, improving the accuracy of AI processing.

[0081] For example, see Figure 1 , Figure 1 This is an exemplary diagram illustrating model training using prior information; such as... Figure 1 As shown, image restoration is performed using the base model 1-1 (e.g., illumination enhancement, deblurring, denoising, rain removal, etc.). Prior information is extracted from the base model 1-1 to obtain the pre-trained model 1-2, which is used to refine the image restoration network 1-3. During testing or inference, the pre-trained model 1-2 and the image restoration network 1-3 are combined to restore the image 1-4 to be restored, resulting in the restored image 1-5.

[0082] See Figure 2 , Figure 2 This is another exemplary diagram illustrating model training using prior information; such as... Figure 2As shown, the high-resolution image 2-11 can be used alone to perform adversarial analysis against the super-resolution image 2-13 of the low-resolution image 2-12 generated by generator 2-21 in discriminator 2-22 to obtain discrimination result 2-31. Furthermore, the image semantics 2-41 of the high-resolution image 2-11 can be extracted, and the high-resolution image 2-11 and the super-resolution image 2-13 can be discriminated against in discriminator 2-23 based on the image semantics 2-41 to obtain discrimination result 2-32. Discriminator 2-23 includes a semantic awareness module 2-231, which enhances the image semantics 2-41. High resolution refers to a resolution higher than a first resolution threshold, and low resolution refers to a resolution lower than a second resolution threshold. Both the first and second resolution thresholds can be set manually or determined based on statistical data; this embodiment does not limit this. Here, the first resolution threshold is greater than or equal to the second resolution threshold.

[0083] because Figure 1 The pre-trained models 1-2, built based on prior information, are used in both testing and inference processes, thus increasing the computational cost of the image restoration network; and because Figure 2 The acquisition of image semantics is based solely on the high-resolution image 2-11 itself, which affects the accuracy of image semantics, thus affecting the accuracy of adversarial training, and consequently the accuracy of the trained image enhancement model.

[0084] Based on this, embodiments of this application provide a training method, image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image enhancement model, which can improve the accuracy of the trained image enhancement model while preserving the computational cost of the trained model. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as robots, smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart in-vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers. It can also be implemented as a server, or a combination of both. The following will describe exemplary applications when the electronic device is implemented as a server or terminal.

[0085] See Figure 3 , Figure 3 This is a schematic diagram of the architecture of the image processing system provided in the embodiments of this application; as shown Figure 3As shown, to support an image processing application, in the image processing system 100, a terminal 600 (terminals 600-1 and 600-2 are shown as examples) connects to a server 200 via a network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the image processing system 100 includes a database 500 for providing data support to the server 200; and... Figure 3 The example shown illustrates a scenario where the database 500 is independent of the server 200. However, the database 500 can also be integrated into the server 200, and this embodiment does not limit this to any particular case.

[0086] Server 200 is used to enhance sample images using an image enhancement model to obtain enhanced sample images; determine the first descriptive text of the label image; extract the label image semantics of the label image based on the first descriptive text; determine the loss value of the target loss function based on the enhanced sample image and the label image semantics; extract the first image features of the enhanced sample image based on the label image semantics; determine the loss value of the target loss function based on the first image features and the target loss function pre-set for the image enhancement model; and train the image enhancement model based on the loss value of the target loss function to obtain the trained image enhancement model. It is also used to receive the image to be enhanced sent by terminal 600 via network 300, enhance the image to be enhanced using the trained image enhancement model to obtain an enhanced image, and then send the enhanced image to terminal 600 via network 300.

[0087] Terminal 600 is used to respond to an image enhancement request, obtain the image to be enhanced as requested in the image enhancement request, send the image to be enhanced to server 200 through network 300, receive the enhanced image sent by server 200 through network 300, and display the enhanced image (graphical interfaces 600-11 and 600-21 are shown as examples).

[0088] It should be noted that the server 200 can also deploy the trained image augmentation model on the terminal 600, so that after the terminal 600 obtains the image to be augmented, it can augment the image to be augmented by the trained image augmentation model deployed on itself, and obtain the augmented image.

[0089] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0090] See Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown... Figure 4 As shown, electronic device 400 includes at least one processor 410, memory 450, at least one network interface 420, and user interface 430. The various components in electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general labeled all buses as Bus System 440.

[0091] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0092] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0093] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0094] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0095] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0096] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0097] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc.

[0098] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0099] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0100] In some embodiments, the apparatus provided in this application can be implemented in software; Figure 4 A training device 455 stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a model enhancement module 4551, a text determination module 4552, a semantic extraction module 4553, a loss determination module 4554, and a model training module 4555. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0101] based on Figure 4 See Figure 5 , Figure 5 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application; as shown Figure 5 As shown, with Figure 4 The difference is, Figure 5 An image processing device 456 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an image acquisition module 4561, a model application module 4562, and a result output module 4563. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0102] In some embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method and image processing method of the image enhancement model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0103] In some embodiments, the terminal or server can implement the training method and image processing method of the image enhancement model provided in this application embodiment by running various computer-executable instructions or computer programs. Here, the implementation of the image processing method is taken as an example for explanation. The computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an image enhancement APP; or it can be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to the browser environment to run. In summary, the above-mentioned computer-executable instructions can be any form of instruction, and the above-mentioned computer program can be any form of application, module, or plugin.

[0104] The training method for the image enhancement model provided in this application will be described below with reference to exemplary applications and implementations of the electronic devices provided in the embodiments of this application. Furthermore, the training method for the image enhancement model provided in the embodiments of this application can be applied to various image enhancement scenarios such as cloud technology, artificial intelligence, smart transportation, medical applications, and automotive applications.

[0105] See Figure 6A , Figure 6A This is a flowchart illustrating the training method of the image enhancement model provided in the embodiments of this application. Figure 1 ;in, Figure 6A The main entity executing each step is the model training device, which will be discussed below. Figure 6A The steps shown are explained.

[0106] Step 101: Obtain sample images and label images.

[0107] It should be noted that the training data used to train the image augmentation model includes sample images and label images of the sample images.

[0108] Step 102: Enhance the sample image using an image enhancement model to obtain the enhanced sample image.

[0109] In this embodiment, after obtaining the image enhancement model to be trained, the model training device begins to execute the training process of the image enhancement model. During the training process, firstly, the model training device acquires sample images for training the image enhancement model; then, the model training device enhances the sample images using the image enhancement model, and the resulting enhancement is referred to as the enhanced sample image.

[0110] It should be noted that the image enhancement model is an artificial neural network model to be trained for enhancing images, such as the image enhancement model MIRNet to be trained, the high-resolution image restoration model Restormer to be trained, the image enhancement model SNR-aware to be trained, the image enhancement model Retinextformer to be trained, the image enhancement model LLCaps to be trained, etc.; the image enhancement model can be a raw neural network model constructed by initializing parameters, or a pre-trained neural network model, or a large model, etc., and this application embodiment does not limit this; in addition, the image enhancement model enhances the sample image from at least one of the following: resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content; and the enhancement magnitude of the image enhancement model on the sample image is determined based on the model parameters included in the image enhancement model for determining the enhancement magnitude. The sample image is an image to be enhanced for training the image enhancement model. Sample images include at least one of the following: images acquired in low-light environments with illumination intensity below a specified intensity, images containing areas to be repaired, images with resolution below a first resolution threshold, images containing noise, etc. The specified illumination intensity can be manually set or determined based on statistical data; this application does not limit this. Enhancement refers to the process of improving image quality by improving the visual effect or the recognizability of specific features, such as contrast enhancement, sharpening, noise reduction, brightness adjustment, color balance, content restoration, etc. The enhanced sample image is a sample image enhanced by the image enhancement model, and the enhancement effect of the enhanced sample image is positively correlated with the accuracy of the image enhancement model; that is, the higher the accuracy of the image enhancement model, the better the enhancement effect of the enhanced sample image; conversely, the lower the accuracy of the image enhancement model, the worse the enhancement effect of the enhanced sample image.

[0111] Step 103: Determine the first description text.

[0112] In this embodiment, the model training device trains the image enhancement model using a supervised method, thereby obtaining the label image of the sample image when acquiring the sample image. Here, the model training device acquires the text-based descriptive data of the label image, and refers to the acquired text-based descriptive data of the label image as the first descriptive text. Therefore, the first descriptive text is used to describe the image content of the label image. The model training device can determine the first descriptive text of the label image using a graph-to-text method, or it can determine the first descriptive text of the label image through an image-text model, or it can be obtained through manual annotation, or a combination of the above, etc. This embodiment does not limit the scope of the method. Among these, image-to-text (IPT) refers to converting images into text descriptions to automatically generate descriptive text for images. In other words, IPT describes images in text form. Here, processing label images using IPT means obtaining a textual description of the label image. Image-to-text models learn the correspondence between images and text to determine the descriptive text for a given image. That is, when a label image is input into an image-to-text model, the output of the image-to-text model for the label image is the first descriptive text, such as the CLIP model. Manual annotation refers to manually determining the descriptive text for a label image, for example, by responding to input operations for the descriptive text of the label image to obtain the input descriptive text.

[0113] It should be noted that a labeled image refers to an enhanced sample image that has been annotated, serving as a reference enhancement result for a pre-determined sample image. "Pre-determined" means before the sample image is enhanced using an image enhancement model. The image enhancement model is trained using the labeled image as a reference enhancement result, considering at least one of the following: resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content. The labeled image can be, for example, the output result obtained from a pre-trained image enhancement model (different from the image enhancement model itself) on a sample image, or it can be the result obtained manually by enhancing the sample image in at least one of the following aspects: resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content. Here, since the model training device trains the image enhancement model in a supervised manner, the training data used to train the model includes both sample images and the labeled results of the sample images, which are called label images. Labeling refers to the process of obtaining reference enhancement results for the sample images, which can be achieved by enhancing the content to be enhanced in the sample images. The labeled enhanced sample images are the reference enhancement results of the sample images and serve as the learning standard for the image enhancement model. The enhanced sample images, on the other hand, are the output of the image enhancement model and are the predicted enhancement results of the sample images; therefore, the two are different. For example, the label images are superior to the enhanced sample images in terms of resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content. Furthermore, the difference between the augmented sample image and the label image is determined based on the model parameters of the image augmentation model. This is because the augmented sample image is the output of the image augmentation model for the sample image, while the label image is the learning object that the image augmentation model uses to augment the sample image. The higher the accuracy of the model parameters of the image augmentation model, the smaller the difference between the augmented sample image and the label image. Conversely, the lower the accuracy of the model parameters of the image augmentation model, the greater the difference between the augmented sample image and the label image. The first descriptive text is textual description data of the label image's content. Textual description data refers to descriptive data in text form. That is, the first descriptive text is the result obtained by describing the content of the label image in text form. It is used to describe the image content in the label image in text form, such as including at least one of the following information: the main body of the label image, the structure of the main body of the label image, the texture of the main body of the label image, the color of the label image, the position of the foreground of the label image, and the size of the main body of the label image, etc. The label image is different from the image creation time and size attributes of the label image described in text form. Therefore, the first descriptive text is a textual description of the label image, which is a representation result of the text type corresponding to the label image. Thus, the label image and the first text description are representation results under different representation types of the same object.In addition, the similarity between the first descriptive text and the label image is greater than a specified similarity threshold, so that the first descriptive text can accurately describe the image content in the label image. The specified similarity threshold, such as 0.99, can be set manually or determined based on statistical data. This application embodiment does not limit this. Furthermore, the calculation of the similarity between the first descriptive text and the label image can be performed by any method of calculating the similarity between two different modal data. This application embodiment does not limit this.

[0114] See Figure 6B , Figure 6B The embodiments provided in this application are related to Figure 6A The corresponding logical flow diagram; such as Figure 6B As shown, the training data for image enhancement model 6-1 includes sample image 6-21 and label image 6-22; the model training device inputs sample image 6-21 into image enhancement model 6-1 for enhancement, resulting in enhanced sample image 6-3, and determines the first descriptive text 6-4 of label image 6-22.

[0115] See Figure 7A , Figure 7A This is a flowchart illustrating the training method of the image enhancement model provided in the embodiments of this application. Figure 2 ;in, Figure 7A The main entity executing each step is the model training device, such as... Figure 7A As shown in the embodiment of this application, step 103 can be implemented through steps 1031 to 1034; that is, the model training device determines the first description text, including steps 1031 to 1034, and each step is described below.

[0116] Step 1031: Determine the first initialization text of the label image and the second initialization text of the auxiliary sample image.

[0117] In this embodiment of the application, the model training device initializes the text description data of the label image, thus obtaining the first initial text; the model training device can also obtain the image of the label image to be enhanced, which is referred to as the auxiliary sample image; the model training device initializes the text description data of the auxiliary sample image, thus obtaining the second initial text.

[0118] It should be noted that the auxiliary sample image is the image to be enhanced to assist the label image in optimizing the first initialization text. Here, the auxiliary sample image can be a sample image or an image to be enhanced that is unrelated to the label image. This application embodiment does not limit this. The first initialization text refers to the initial text description data of the label image. That is, the first initialization text is the initial text description of the label image, which is the initial text representation result of the image content of the label image. The second initialization text is the initial text description data of the auxiliary sample image. That is, the second initialization text is the initial text description of the auxiliary sample image, which is the initial text representation result of the image content of the auxiliary sample image.

[0119] Step 1032: Determine the first similarity between the label image and the first initialization text, and the second similarity between the auxiliary sample image and the second initialization text.

[0120] In this embodiment, the model training device determines the similarity between the label image and the first initialization text, thus obtaining a first similarity; and determines the similarity between the auxiliary sample image and the second initialization text, thus obtaining a second similarity. Here, the model training device can determine the first similarity by comparing the features of the label image and the features of the first initialization text, and determine the second similarity by comparing the features of the auxiliary sample image and the features of the second initialization text; and the first and second similarities can be determined using at least one of the following methods: cosine similarity, Euclidean distance, and Manhattan distance, etc.

[0121] It should be noted that the first similarity score represents the similarity between the label image and the first initialization text. Since the first initialization text is the initial description text of the label image, whether the label image and the first initialization text are similar is yet to be determined. Therefore, the model training device calculates the similarity between the label image and the first initialization text, thus obtaining the first similarity score. Here, the higher the first similarity score, the more similar the label image and the first initialization text are; conversely, the lower the first similarity score, the greater the difference between the label image and the first initialization text. The second similarity score represents the similarity between the auxiliary sample image and the second initialization text. Since the second initialization text is the initial description text of the auxiliary sample image, whether the auxiliary sample image and the second initialization text are similar is yet to be determined. Therefore, the model training device calculates the similarity between the auxiliary sample image and the second initialization text, thus obtaining the second similarity score. Here, the higher the second similarity score, the more similar the auxiliary sample image and the second initialization text are; conversely, the lower the second similarity score, the greater the difference between the auxiliary sample image and the second initialization text.

[0122] Step 1033: Based on the first similarity and the second similarity, determine the loss value of the second loss function between the first initialization text and the label image.

[0123] In this embodiment, the model training device determines the loss value between the first initialization text and the label image based on the first similarity and the second similarity, thus obtaining the loss value of the second loss function. The loss value of the second loss function characterizes the degree of difference between the first initialization text and the label image. Here, the model training device can use the cross-entropy loss calculation method to determine the loss value of the second loss function based on the first and second similarities; that is, the model training device first obtains the sum of the first and second similarities, then determines the ratio of the first similarity to the sum as the first prediction probability, which is the prediction probability of the label image, representing the probability of predicting the label image as a positive label, where a positive label refers to the reference augmentation result; the cross-entropy loss is calculated based on the prediction probability and the true label of the label image (e.g., 1), and the calculation result is called the loss value of the second loss function. The second loss function is used to adjust the loss function of the first initial text, and is used to measure the accuracy of the first initial text, providing direction for the optimization of the first initial text; that is, the first initial text is optimized by minimizing the loss value of the second loss function; in addition, the loss value of the second loss function is negatively correlated with the accuracy of the first initial text.

[0124] It should be noted that the calculation of cross-entropy loss is achieved through the cross-entropy loss function; the cross-entropy loss function is also known as the log loss function, which is used to measure the difference between the predicted probability and the true label. The difference between the predicted probability and the true label can be achieved by calculating the negative log-likelihood between the predicted probability and the true label, as shown in the following formula (1).

[0125] Step 1034: Adjust the first initialization text based on the loss value of the second loss function to obtain the first description text.

[0126] In this embodiment, the model training device adjusts the first initialization text based on the loss value of the second loss function. The adjustment direction is to reduce the loss value of the second loss function to decrease the distance between the adjusted first initialization text and the label image. The adjusted first initialization text is the first descriptive text. Here, the model training device can iteratively adjust the first initialization text based on the loss value of the second loss function. When the first iteration termination condition is met, the obtained adjustment result is determined as the first descriptive text. The first iteration termination condition can be reaching a first accuracy index threshold, exceeding a first iteration number threshold, exceeding a first iteration duration threshold, or a combination of the above, etc. This embodiment does not limit this. The first accuracy index threshold, the first iteration number threshold, and the first iteration duration threshold can be manually set or determined based on statistical data. This embodiment does not limit this.

[0127] See Figure 7B , Figure 7B This is a schematic diagram of the logical flow for determining the first descriptive text provided in an embodiment of this application; as follows: Figure 7B As shown, the model training device initializes the text description of the label image 7-11 to obtain the first initialized text 7-21, and initializes the text description of the auxiliary sample image 7-12 to obtain the second initialized text 7-22. The first similarity 7-31 between the label image 7-11 and the first initialized text 7-21, and the second similarity 7-32 between the auxiliary sample image 7-12 and the second initialized text 7-22 are determined. Based on the first similarity 7-31 and the second similarity 7-32, the loss value 7-4 of the second loss function between the first initialized text 7-21 and the label image 7-11 is determined. Finally, the first initialized text 7-21 is adjusted based on the loss value 7-4 of the second loss function to obtain the first descriptive text 7-5.

[0128] In this embodiment, the model training device can use an image-text model to determine the first descriptive text of the label image. At this time, the model training device obtains the features of the label image and the auxiliary sample image through the image encoder in the image-text model, and obtains the features of the first initialization text and the second initialization text through the text encoder in the image-text model, to calculate the first similarity and the second similarity. The image encoder is used to extract image features, and the text encoder is used to extract text features. The features of the first initialization text are obtained by feature extraction from the first initialization text, and the features of the second initialization text are obtained by feature extraction from the second initialization text. Furthermore, the model training device obtains the first similarity by calculating the similarity between the features of the label image and the features of the first initialization text; and obtains the second similarity by calculating the similarity between the features of the auxiliary sample image and the features of the second initialization text. It is readily understood that the model training device represents the similarity between the label image and the first initialization text through the similarity between the features of the label image and the features of the first initialization text, and represents the similarity between the auxiliary sample image and the second initialization text through the similarity between the features of the auxiliary sample image and the features of the second initialization text.

[0129] In this embodiment, the model training device adjusts the first initialization text based on the loss value of the second loss function to obtain the first descriptive text, which is achieved while freezing the image-text model; that is, the model parameters in the image-text model are fixed, and the first initialization text is adjusted during backpropagation based on the loss value of the second loss function. Furthermore, when adjusting the first initialization text based on the loss value of the second loss function, the model training device also determines the loss value of the third loss function between the second initialization text and the auxiliary sample image based on the first similarity and the second similarity, and adjusts the second initialization text based on the loss value of the third loss function; here, the adjustment of the first initialization text and the second initialization text are performed synchronously, and the adjustment of the second initialization text is used to implement the second and subsequent adjustments of the first initialization text. Here, the process of obtaining the loss value of the third loss function is similar to that of the second loss function. The model training device can use the cross-entropy loss calculation method to determine the loss value of the third loss function based on the first and second similarities. That is, the model training device first obtains the sum of the first and second similarities, then obtains the third similarity between the auxiliary sample image and the first initialization text, and determines the ratio of the third similarity to the sum as the second prediction probability. Based on the second prediction probability and the true label of the auxiliary sample image (e.g., 0), the cross-entropy loss is calculated, and the calculation result is called the loss value of the third loss function.

[0130] It should be noted that the third loss function is used to adjust the loss function of the second initialization text, and is used to measure the accuracy of the second initialization text, providing direction for the optimization of the second initialization text; that is, the second initialization text is optimized by minimizing the loss value of the third loss function; in addition, the loss value of the third loss function is negatively correlated with the accuracy of the second initialization text.

[0131] It is understandable that, since the matching relationship between the image and the text is obtained in advance before obtaining the first descriptive text, the matching degree between the image and the text can be accurately determined. Then, based on the pre-determined matching relationship between the image and the text, the first similarity between the label image and the first initialized text, and the second similarity between the auxiliary sample image and the second initialized text can be determined. Then, the first initialized text can be adjusted based on the first similarity and the second similarity, which can improve the accuracy of the adjusted first descriptive text.

[0132] Step 104: Extract the semantic meaning of the label image based on the first descriptive text.

[0133] In this embodiment, the model training device adjusts the semantics of the label image based on the first descriptive text, thus obtaining the label image semantics. The label image semantics is used to characterize the semantics of the label image, and it is obtained by combining the first descriptive text and the label image. Here, since the first descriptive text and the label image are representations of the same object under two different representation types, the semantic features of the first descriptive text represent text-dimensional features, while the semantic features of the label image represent image-dimensional features. The model training device can fuse the semantic features of the first descriptive text with the semantic features of the label image to adjust the semantics of the label image based on the first descriptive text, thereby obtaining the label image semantics. In other words, the label image semantics is obtained by fusing the semantics of the first descriptive text and the semantics of the label image itself; it is the result of fusing the semantics of the first descriptive text and the label image, two modalities of data.

[0134] It is understandable that the semantics of the label image extracted from the first descriptive text is the final semantics of the label image obtained by combining the semantics of the first descriptive text and the semantics of the label image; thus, compared with directly extracting the semantics of the label image, it can improve the comprehensiveness and richness of the semantics of the label image.

[0135] See also Figure 7A In this embodiment of the application, step 104 can be implemented by steps 1041 to 1043; that is, the model training device extracts the semantic meaning of the label image based on the first descriptive text, including steps 1041 to 1043, and each step is described below.

[0136] Step 1041: Extract the first text features of the first descriptive text.

[0137] It should be noted that the model training device extracts features from the first descriptive text, and the extracted features are the first text features; therefore, the first text features are the features of the first descriptive text. Here, a trained text encoder can be used to extract the first text features, or a specified text encoding method can be used to obtain the first text features, etc., and this application embodiment does not limit this.

[0138] Step 1042: Extract the label image features from the label image.

[0139] It should be noted that the model training device extracts features from the label image, and the extracted features are the label image features; therefore, the label image features are the characteristics of the label image. Here, a trained image encoder can be used to extract label image features, or a specified image encoding method can be used to obtain label image features, etc., and this application embodiment does not limit this.

[0140] In the embodiments of this application, steps 1041 and 1042 are not ordered in any particular order. Step 1041 can be executed first and then step 1042, or step 1042 can be executed first and then step 1041, or steps 1041 and 1042 can be executed simultaneously, or steps 1041 and 1042 can be executed alternately, etc. The embodiments of this application do not limit this.

[0141] Step 1043: Fuse the first text features and the label image features to obtain the label image semantics.

[0142] It should be noted that the model training device enhances the semantics of the label image by fusing the first text features and the label image features to obtain the semantic meaning of the label image. Here, the model training device can achieve the fusion of the first text features and the label image features in the following ways: affine transformation, weighted summation, concatenation, accumulation, etc. Among them, affine transformation is a linear transformation, including matrix multiplication and bias term addition. Matrix multiplication is achieved by multiplying the matrix vector with the input, and bias term addition is achieved by adding the bias term to the input as an independent term. In addition, the matrix vector represents the parameter matrix connecting two adjacent layers in the neural network; the bias term is a parameter vector attached to each output neuron, representing a fixed offset introduced by the neural network when performing linear combination. Therefore, the bias term can be a constant added to the activation function of each neuron to increase the flexibility of the affine transformation.

[0143] In this embodiment of the application, when the model training device uses affine transformation to fuse the first text features and the label image features, in step 1043 of this embodiment of the application, the model training device fuses the first text features and the label image features to obtain the label image semantics, including: the model training device first performs affine transformation on the first text features to obtain scaling parameters and bias terms; then determines the feature fusion result of the scaling parameters and the label image features; finally, the label image features, the bias terms and the feature fusion result are combined to form the label image semantics.

[0144] It should be noted that the model training device performs an affine transformation on the first text features. The obtained affine transformation parameters include scaling parameters and bias terms. The scaling parameters, used to adjust the label image features, are matrix vectors in the affine transformation, representing the weight matrix corresponding to the label image features, and are used for multiplication with the label image features. The bias term is additional information about the label image features, representing the offset of the label image features, and is added to the product of the label image features and the scaling parameters. Here, the model training device can implement the affine transformation of the first text features using a multilayer perceptron (MLP). The model training device can obtain the feature fusion result by performing a dot product between the scaling parameters and the label image features; and the model training device can combine the label image, bias term, and feature fusion result using at least one of the following methods: accumulation, weighted summation, concatenation, etc.

[0145] See Figure 7C , Figure 7C This is a schematic diagram of the logical flow for determining the semantics of a label image provided in an embodiment of this application; as follows: Figure 7C As shown, the model training device extracts the first text feature 7-62 of the first descriptive text 7-61 and the label image feature 7-64 of the label image 7-63; then, it performs an affine transformation on the first text feature 7-62 to obtain the scaling parameter 7-66 and the bias term 7-67 of the label image feature 7-64; finally, it fuses the scaling parameter 7-66 and the label image feature 7-64 to obtain the feature fusion result 7-68; and the label image feature 7-64, the bias term 7-67 and the feature fusion result 7-68 are combined to form the label image semantics 7-69.

[0146] Step 105: Extract the first image features of the enhanced sample image based on the semantics of the labeled image.

[0147] It should be noted that since the semantics of the labeled image is used to represent the semantics of the labeled image, and is a pre-determined semantics of the reference augmentation result, the model training device extracts features of the augmented sample image in the semantic dimension of the reference augmented image based on the semantics of the labeled image. These extracted features are referred to as the first image features. Here, the model training device can perform feature extraction on the augmented sample image once based on the semantics of the labeled image to obtain the first image features. Alternatively, the model training device can iteratively extract the first image features of the augmented sample image based on the semantics of the labeled image; that is, the model training device extracts intermediate image features of the augmented sample image based on the semantics of the labeled image, then extracts features of the intermediate image features based on the semantics of the labeled image, and so on iteratively until the end, with the final extracted features referred to as the first image features.

[0148] See Figure 7D , Figure 7D This is a schematic diagram of the logic flow for obtaining the first image features provided in an embodiment of this application; as shown below. Figure 7D As shown, the model training device extracts the first image features 7-73 of the enhanced sample image 7-72 ​​based on the semantics of the labeled image 7-71.

[0149] See Figure 8A , Figure 8A This is a schematic diagram of the process for obtaining the first image features provided in an embodiment of this application; wherein, Figure 8A The main entity executing each step is the model training device, such as... Figure 8A As shown in the embodiment of this application, step 105 can be implemented through steps 1051 to 1053; that is, the model training device extracts the first image features of the enhanced sample image based on the semantics of the label image, including steps 1051 to 1053. Each step is described below.

[0150] Step 1051: Determine the target query features based on the first text features and the semantics of the label image.

[0151] It should be noted that the model training device adjusts the semantics of the labeled image based on the first textual features of the first descriptive text, and the adjusted semantics of the labeled image is called the target query feature. The target query feature is the query (Q) of the augmented sample image, representing the information feature to be queried in the augmented sample image.

[0152] See also Figure 8A In this embodiment of the application, step 1051 can be implemented through steps 10511 and 10512; that is, the model training device determines the target query features based on the first text features and the semantics of the label image, including steps 10511 and 10512. Each step is described below.

[0153] Step 10511: Enhance the semantics of the label image to obtain the semantics of the target image.

[0154] It should be noted that the model training device can enhance the semantics of the label image by normalizing the semantics of the label image and then performing convolutional modulation on the normalized result of the semantics of the label image, or it can directly enhance the semantics of the label image by performing convolutional modulation on the semantics of the label image. This application embodiment does not limit this; here, the enhanced semantics of the label image obtained is the semantics of the target image.

[0155] In this embodiment, convolutional modulation refers to the process of enhancing the expressive power of features by convolving features, including determining feature weights, convolving features, and fusing weights with convolution results. When the model training device enhances the semantics of the labeled image through convolutional modulation, in step 10511 of this embodiment, the model training device enhances the semantics of the labeled image to obtain the target image semantics, including: the model training device first determines the image semantic weights of the labeled image semantics; then convolves the labeled image semantics to obtain image semantic features; finally, the image semantic weights and image semantic features are fused to obtain the target image semantics.

[0156] It should be noted that image semantic weights are used to enhance the semantics of labeled images. The model training device can determine the image semantic weights of labeled images using an M*M deep convolutional layer, where M is an integer greater than 1; it can also perform convolution on the semantics of labeled images using a 1*1 convolutional layer; here, the model training device can fuse image semantic weights and image semantic features through dot product, and the fused result of image semantic weights and image semantic features is the target image semantics.

[0157] It is understandable that by enhancing the semantics of the labeled image to obtain the semantics of the target image, the information in the labeled image semantics that is effective for image enhancement can be highlighted and enriched, so that the semantics of the target image can more accurately represent the semantics of the labeled image than the semantics of the labeled image, thereby improving the accuracy of the semantics of the target image.

[0158] Step 10512: Fuse the first text features and the target image semantics to obtain the target query features.

[0159] It should be noted that the process of fusing the first text features and the target image semantics by the model training device is similar to the process of fusing the first text features and the label image features. That is, after obtaining the scaling parameters and bias terms through affine transformation of the first text features, the model training device obtains the fusion result of the scaling parameters and the target image semantics, and then combines the target image semantics, the bias terms, and the fusion result to achieve the fusion of the first text features and the target image semantics. Specifically, the process of obtaining the fusion result of the scaling parameters and the target image semantics by the model training device is similar to the process of obtaining the feature fusion result; that is, the model training device obtains the fusion result of the scaling parameters and the target image semantics by multiplying them. Furthermore, the process of combining the target image semantics, the bias terms, and the fusion result by the model training device is similar to the process of combining the label image features, the bias terms, and the feature fusion result; that is, the model training device combines the target image semantics, the bias terms, and the fusion result using at least one of the following methods: accumulation, weighted summation, concatenation, etc.

[0160] In this embodiment, the model training device can directly use the combination of target image semantics, bias terms, and fusion results as target query features. Alternatively, it can process the combination of target image semantics, bias terms, and fusion results through layer normalization and use the processing results as target query features. This embodiment does not limit the scope of the application.

[0161] It is understandable that, since the first text feature is a semantic dimension different from that of the label image, fusing the first text feature and the semantic dimension of the target image to obtain the target query feature can improve the comprehensiveness of the target query feature in semantic context representation, thereby improving the accuracy of the target query feature.

[0162] Step 1052: Determine the first key feature and the first value feature of the enhanced sample image.

[0163] It should be noted that the model training device determines the key and value based on the features of the augmented sample image. The key (Key, K) of the determined features of the augmented sample image is called the first key feature, and the value (Value, V) of the determined features of the augmented sample image is called the first value feature. In other words, the first key feature is the information feature in the augmented sample image to be compared with the target query feature, and the first value feature is the content feature in the augmented sample image corresponding to the first key feature.

[0164] In this embodiment of the application, the model training device can obtain the first key feature by obtaining the product of the key parameter and the feature of the enhanced sample image; and can obtain the first value feature by obtaining the product of the value parameter and the feature of the enhanced sample image; wherein the key parameter and the value parameter can be learned parameters.

[0165] Step 1053: Perform attention processing on the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image.

[0166] In this embodiment, when the model training device performs attention processing on the target query feature, the first key feature, and the first value feature, it first obtains the product between the target query feature and the first key feature; then it obtains the ratio of the product to a specified projection dimension, which can be manually set, determined based on statistical data, or learned, etc., and this embodiment does not limit this; next, it normalizes the ratio, for example, by using a normalization function (e.g., the Softmax function); finally, it obtains the product of the normalization result and the first value feature, and uses the product as the first image feature, or continues to perform attention processing based on the product to obtain the first image feature.

[0167] It should be noted that when the model training device performs attention processing on the target query feature, the first key feature, and the first value feature, it can first obtain the product of the target query feature and the first key feature, then obtain the normalized result of the product, and finally use the product of the normalized result and the first value feature as the attention processing result. The attention processing result is then concatenated with the features of the enhanced sample image through a channel concatenation process, thus obtaining the first image feature of the enhanced sample image. Here, the model training device can directly use the result of concatenating the attention processing result with the features of the enhanced sample image through a channel concatenation process as the first image feature; alternatively, it can iteratively perform attention processing on the result of concatenating the attention processing result with the features of the enhanced sample image through a channel concatenation process until a specified number of iterations is reached, at which point the attention processing iteration stops, and the final channel concatenation result is used as the first image feature. The specified number of iterations, for example, is 3 times, which can be manually set or determined based on statistical data; this embodiment does not limit this.

[0168] In this embodiment of the application, after obtaining the result of one attention processing of the target query feature, the first key feature, and the first value feature, if attention processing is performed again on the result of one attention processing, then in step 1053, the model training device performs attention processing on the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image. This includes: the model training device performs attention processing on the target query feature, the first key feature, and the first value feature to obtain the intermediate image feature of the enhanced sample image; then determines the second key feature and the second value feature of the intermediate image feature; finally, attention processing is performed on the target query feature, the second key feature, and the second value feature to obtain the first image feature of the enhanced sample image.

[0169] It should be noted that the intermediate image features of the enhanced sample image refer to the result obtained from one attention processing step of the target query feature, the first key feature, and the first value feature. When the model training device continues to perform attention processing on the intermediate image features, it obtains the key of the intermediate image features, which is the second key feature, and obtains the value of the intermediate image features, which is the second value feature. Thus, the second key feature is the information feature in the intermediate image features to be compared with the target query feature, and the second value feature is the content feature in the intermediate image features corresponding to the second key feature. The process by which the model training device determines the second key feature and the second value feature of the intermediate image features is similar to the process of determining the first key feature and the first value feature of the enhanced sample image; the second key feature can be obtained by obtaining the product of the key parameter and the intermediate image feature; the second value feature can be obtained by obtaining the product of the value parameter and the intermediate image feature. Here, the model training device can determine the first image feature based on the attention processing result of the target query feature, the second key feature, and the second value feature, or it can determine the first image feature based on the iterative attention processing result of the target query feature, the second key feature, and the second value feature. This application embodiment does not limit this.

[0170] In this embodiment, when the model training device performs attention processing on the target query feature, the second key feature, and the second value feature, it first obtains the product between the target query feature and the second key feature; then it obtains the ratio of the product to the specified projection dimension; next, it normalizes the ratio, for example, by using a normalization function; finally, it obtains the product of the normalization result and the second value feature, so as to use the product as the first image feature, or to continue to perform attention processing based on the product to obtain the first image feature.

[0171] In this embodiment, the model training device can integrate a discriminator into the image enhancement model to be trained through adversarial training, so as to discriminate the enhancement quality of the enhanced sample image through the discriminator. At this time, the model training device extracts the first image features of the enhanced sample image based on the semantics of the labeled image, which is achieved through the discriminator; and the discriminator and the image enhancement model are trained collaboratively.

[0172] See Figure 8B , Figure 8B The embodiments provided in this application are related to Figure 8A The corresponding logical flow diagram; such as Figure 8B As shown, the model training device determines the target query feature 8-31 based on the first text feature 8-1 and the label image semantics 8-2. In determining the target query feature 8-31, the model training device first enhances the label image semantics 8-2 to obtain the target image semantics 8-4, and then fuses the first text feature 8-1 and the target image semantics 8-4 to obtain the target query feature 8-31. Afterwards, the model training device determines the first key feature 8-32 and the first value feature 8-33 of the enhanced sample image 8-5; attention processing is then applied to the target query feature 8-31, the first key feature 8-32, and the first value feature 8-33 to obtain the first image feature 8-6 of the enhanced sample image 8-5.

[0173] Step 106: Based on the first image features and the target loss function pre-set for the image enhancement model, determine the loss value of the target loss function.

[0174] In this embodiment, since the target loss function is used to characterize the mapping relationship between the first image features and the loss value of the target loss function, the model training device can calculate the loss value by substituting the first image features into the target loss function. Here, when training the image enhancement model, if adversarial learning is used, the model training device can pre-set the first adversarial loss function as the target loss function. Furthermore, since image enhancement involves the viewpoint dimension, the perceptual loss function can also be pre-set as the target loss function. And since the image enhancement model is the baseline model to be trained, it has its own corresponding loss function, so the loss function corresponding to the image enhancement model itself can also be set as the target loss function. Therefore, the model training device can pre-set at least one of the loss function corresponding to the image enhancement model itself and the perceptual loss function, as well as the first adversarial loss function, as the target loss function.

[0175] It should be noted that the sample images and label images are training data used to train the image enhancement model. The enhanced sample image is the predicted enhancement result corresponding to the sample image obtained by the image enhancement model. The label image is the reference enhancement result of the sample image, and the label image semantics is used to characterize the semantics of the label image. Therefore, based on the first image features extracted from the label image semantics and the target loss function, the difference between the enhanced sample image and the label image can be determined. The loss value of the target loss function is negatively correlated with the accuracy of the image enhancement model in enhancing the sample image. That is, the larger the loss value of the target loss function, the worse the enhancement result of the enhanced sample image in at least one of the following aspects: resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content.

[0176] See also Figure 7A In this embodiment of the application, step 106 can be implemented by steps 1061 to 1062; that is, the model training device determines the loss value of the target loss function based on the first image features and the target loss function pre-set for the image enhancement model, including steps 1061 to 1062. Each step will be described below.

[0177] Step 1061: Determine the loss value of the first adversarial loss function based on the first image features and the first adversarial loss function.

[0178] It should be noted that the model training device, by determining the image enhancement score corresponding to the first image feature, can obtain the corresponding loss value of the adversarial loss function, which is the loss value of the first adversarial loss function. The first adversarial loss function is the loss function used to train the image enhancement model, measuring its accuracy and providing direction for training; that is, training the image enhancement model by minimizing the loss value of the first adversarial loss function. Furthermore, the first adversarial loss function characterizes the mapping relationship between the first image feature and the loss value of the first adversarial loss function in adversarial learning mode. Adversarial learning mode refers to the model training method through adversarial learning. The loss value of the first adversarial loss function is negatively correlated with the accuracy of the image enhancement model, and the target loss function includes the first adversarial loss function.

[0179] In this embodiment of the application, the model training device determines the loss value of the first adversarial loss function based on the first image features and the first adversarial loss function, including: the model training device first determines the first type label of the enhanced sample image; and then determines the loss value of the first adversarial loss function based on the first type label, the first image features, and the first adversarial loss function.

[0180] It should be noted that the first type of label is the true label of the augmented sample image. Because the image augmentation model aims to obtain the most accurate augmented sample image possible, the label of the augmented sample image is the true label for the image augmentation model, for example, denoted as 1. Here, the first type of label is the label of the augmented sample image relative to the image augmentation model. The model training device calculates the loss value based on this first type of label and the first image features, for example, by calculating cross-entropy loss, and the calculated loss value is called the loss value of the first adversarial loss function.

[0181] Step 1062: Determine the loss value of the target loss function based on the loss value of the first adversarial loss function.

[0182] In this embodiment, the model training device can use the loss value of the first adversarial loss function as the loss value of the target loss function, in which case the first adversarial loss function is the target loss function; alternatively, it can combine the loss value of the first adversarial loss function with the loss values ​​of other loss functions to obtain the loss value of the target loss function, in which case the combination result of the first adversarial loss function and other loss functions is the target loss function; this embodiment does not limit the scope of the application.

[0183] In this embodiment, when the model training device combines the loss value of the first adversarial loss function with the loss values ​​of other loss functions to obtain the loss value of the target loss function, the model training device determines the loss value of the target loss function based on the loss value of the first adversarial loss function. This includes: the model training device first determines the loss value of the first loss function based on the enhanced sample image, the label image, and the first loss function; then, based on the loss value of the first loss function and the loss value of the first adversarial loss function, it determines the loss value of the target loss function. Here, the loss value of the first adversarial loss function is the loss value of other loss functions combined with the loss value of the first adversarial loss function.

[0184] It should be noted that the target loss function also includes a first loss function, which includes at least one of the loss function corresponding to the image augmentation model and the perceptual loss function. The loss value of the first loss function includes at least one of the model loss value and the perceptual loss value. The model loss value is calculated using the loss function corresponding to the image augmentation model. Since each image augmentation model to be trained includes its own corresponding loss function, such as mean squared error loss, regularization loss, norm loss, etc., the image augmentation model also includes its own corresponding loss function. Using the loss function corresponding to the image augmentation model, a loss value of the augmented sample image can be calculated, which is the model loss value. The perceptual loss value is calculated using the perceptual loss function, which means that the perceptual loss value is obtained by calculating the difference between the augmented sample image and the labeled image in the feature space. The perceptual loss function is used to determine the visual quality of the image from the perspective of perceptual quality. The perceptual loss value can be calculated using a pre-trained neural network to calculate the difference between the sample image and the augmented sample image. Specifically, a pre-trained convolutional neural network can be used to extract features from both the sample image and the augmented sample image. The perceptual loss value is then determined based on the difference between these features. For example, the difference can be obtained by calculating the Euclidean distance or Manhattan distance between the features of the sample image and the augmented sample image. The perceptual loss value minimizes the distance between the sample image and the augmented sample image in the feature space.

[0185] It should also be noted that both the model loss value and the perceptual loss value are calculated based on the augmented sample images and the labeled images; the only difference is that the loss functions used to calculate the model loss value and the perceptual loss value are different.

[0186] Here, the combined result of the loss value of the first loss function and the loss value of the first adversarial loss function is the loss value of the target loss function; when the loss value of the first loss function includes the model loss value and the perceptual loss value, the loss value of the target loss function is the combined result of the loss value of the first adversarial loss function, the model loss value, and the perceptual loss value; when the loss value of the first loss function includes the model loss value, the loss value of the target loss function is the combined result of the loss value of the first adversarial loss function and the model loss value; when the loss value of the first loss function includes the perceptual loss value, the loss value of the target loss function is the combined result of the loss value of the first adversarial loss function and the perceptual loss value.

[0187] In the embodiments of this application, the model training device may combine the loss value of the first loss function and the loss value of the first adversarial loss function in at least one of the following ways: accumulation, weighted summation, averaging, selection of the maximum value, etc.

[0188] See also Figure 7D The model training device calculates the loss value 7-74 of the first adversarial loss function corresponding to the first image feature 7-73. This loss value 7-74 can be directly used as the loss value 7-75 of the target loss function. Alternatively, the model loss value 7-76 can be calculated using the loss function corresponding to the image enhancement model. Based on the model loss value 7-76 and the loss value 7-74 of the first adversarial loss function, the loss value 7-75 of the target loss function can be determined. Furthermore, the perceptual loss value 7-78 of the enhanced sample image 7-72 ​​and the label image 7-77 in the feature space can be calculated. Based on the perceptual loss value 7-78 and the loss value 7-74 of the first adversarial loss function, the loss value 7-75 of the target loss function can be determined. Finally, the loss value 7-75 of the target loss function can be determined based on the model loss value 7-76, the perceptual loss value 7-78, and the loss value 7-74 of the first adversarial loss function.

[0189] Understandably, since the model loss value is calculated using the loss function corresponding to the image augmentation model, the perceptual loss value is obtained by calculating the difference between the augmented sample image and the label image in the feature space, and the loss value of the first adversarial loss function represents the augmentation quality score of the augmented sample image; by determining the final target loss function's loss value using at least one of the model loss value and the perceptual loss value, as well as the loss value of the first adversarial loss function, the accuracy of the target loss function's loss value can be improved; furthermore, model training based on the loss value of the target loss function can be performed accurately, thereby improving the convergence speed of model training.

[0190] Step 107: Train the image augmentation model based on the loss value of the target loss function to obtain the trained image augmentation model.

[0191] In this embodiment, the target loss function is a loss function used to train the image enhancement model, measuring the enhancement accuracy of the image enhancement model and providing direction for its training. Specifically, the image enhancement model is trained by minimizing the loss value of the target loss function to improve its enhancement accuracy in at least one of resolution, contrast, texture, content to be sharpened, noise, brightness, color, and content. Furthermore, the loss value of the target loss function is negatively correlated with the accuracy of the image enhancement model. After obtaining the loss value of the target loss function, the model training device performs backpropagation on the image enhancement model based on this loss value to adjust the model parameters and train the image enhancement model. During backpropagation, the model training device calculates the gradient of the output layer of the image enhancement model based on the loss value of the target loss function. This gradient represents the influence of the output of each neuron in the output layer on the loss value of the target loss function. Then, the parameters of the output layer are adjusted based on this gradient, and the gradient is passed to the layer above the output layer to continue parameter adjustment until the parameters of all layers of the image enhancement model are adjusted. The training process of the image enhancement model can be iterative. Training ends when the second iteration termination condition is met, and the image enhancement model trained in the last iteration is determined as the trained image enhancement model. The second iteration termination condition can be reaching a second accuracy threshold, exceeding a second iteration count threshold, exceeding a second iteration duration threshold, or a combination of these, etc. This application embodiment does not limit this. The second accuracy threshold, the second iteration count threshold, and the second iteration duration threshold can be manually set or determined based on statistical data; this application embodiment does not limit this.

[0192] See also Figure 6B The model training device extracts the semantic information of the label image 6-22 based on the first descriptive text 6-4; and determines the loss value 6-6 of the target loss function based on the difference between the enhanced sample image 6-3 and the semantic information of the label image 6-5; finally, the image enhancement model 6-1 is trained based on the loss value 6-6 of the target loss function to obtain the trained image enhancement model 6-7.

[0193] In this embodiment, the model training device is further used to train the discriminator to obtain an intermediate discriminator; the intermediate discriminator is used to assist in the training of the image enhancement model during the training process. Therefore, step 105, in which the model training device trains the image enhancement model based on the loss value of the target loss function to obtain the trained image enhancement model, includes: the model training device first trains the image enhancement model based on the loss value of the target loss function to obtain an intermediate image enhancement model; then, it trains the intermediate image enhancement model through the intermediate discriminator to obtain the trained image enhancement model.

[0194] It should be noted that the model training device refers to the image enhancement model that has completed one training iteration as the intermediate image enhancement model, and the discriminator that has completed one training iteration as the intermediate discriminator. Here, the model training device trains the intermediate image enhancement model through the intermediate discriminator to obtain the trained image enhancement model, including: the model training device enhances the sample image through the intermediate image enhancement model to obtain a new enhanced sample image; the intermediate discriminator extracts the third image feature of the new enhanced sample image based on the semantics of the label image, and trains the intermediate image enhancement model based on the adversarial loss value corresponding to the third image feature. Correspondingly, the model training device extracts the fourth image feature of the label image based on the semantics of the label image through the intermediate discriminator, and trains the intermediate discriminator based on the adversarial loss value corresponding to the third and fourth image features. This process is iterated until the iterative training of the image enhancement model is completed, resulting in the trained image enhancement model. The process of obtaining the third image feature is similar to the process of obtaining the first image feature, and the process of obtaining the fourth image feature is similar to the process of obtaining the second image feature; these will not be repeated in the embodiments of this application.

[0195] It is understandable that after obtaining the enhanced sample image of the sample image through the image enhancement model, the label image semantics of the label image is extracted based on the first descriptive text corresponding to the label image of the sample image, and the first image features of the enhanced sample image are extracted based on the label image semantics. Based on the first image features and the target loss function pre-set for the image enhancement model, the loss value used to train the image enhancement model is determined to achieve the training of the image enhancement model. In the training process of the image enhancement model, the first image feature used to determine the loss value of the target loss function is extracted based on the semantics of the label image. This semantics is extracted from the label image based on the first descriptive text, which represents the textual dimension of the label image, differing from the label image's information in another dimension. Therefore, extracting the semantics of the label image based on the first descriptive text is a process of determining the semantics of the label image through information from different dimensions. This improves the comprehensiveness of the label image semantics in terms of feature dimensions, thus enhancing its accuracy in representing the reference enhancement result. Consequently, extracting the first image feature used to calculate the loss value of the target loss function based on the semantics of the label image improves the accuracy of the target loss function's loss value. Furthermore, training the image enhancement model based on this target loss function's loss value improves the accuracy of the trained image enhancement model.

[0196] In this embodiment of the application, the training method of the image enhancement model further includes a process of training a discriminator. The process of training the discriminator includes: the model training device first extracts the second image features of the label image based on the semantics of the label image through the discriminator; then, based on the second image features and the first image features, determines the loss value of the second adversarial loss function; finally, the discriminator is trained based on the loss value of the second adversarial loss function to obtain an intermediate discriminator.

[0197] It should be noted that the process by which the model training device extracts the second image features of the label image based on the semantics of the label image through the discriminator is similar to the process by which the discriminator extracts the first image features of the enhanced sample image based on the semantics of the label image. That is, the model training device performs the following processing through the discriminator to obtain the second image features: determining the third key feature and the third value feature of the label image; and performing attention processing on the target query feature, the third key feature, and the third value feature to obtain the second image features. Since the process of obtaining the second image features is similar to that of obtaining the first image features, it will not be described again in this embodiment. Furthermore, the third key feature is the information feature in the label image to be compared with the target query feature, and the third value feature is the content feature in the label image corresponding to the third key feature. The model training device can obtain the third key feature by obtaining the product of the key parameter and the features of the label image; and it can obtain the third value feature by obtaining the product of the value parameter and the features of the label image. Here, when the model training device performs attention processing on the target query feature, the third key feature, and the third value feature, it first obtains the product between the target query feature and the third key feature; then it obtains the ratio of this product to the specified projection dimension; next, it normalizes this ratio, for example, by normalizing the ratio using a normalization function; finally, it obtains the product of the normalization result and the third value feature, so as to use this product as the second image feature, or to continue to perform attention processing based on this product to obtain the second image feature.

[0198] It should also be noted that, since the discriminator is used to distinguish between the labeled image and the augmented sample image, its training is based on the second image features of the labeled image and the first image features of the augmented sample image. Therefore, the model training device determines the adversarial loss value corresponding to the discriminator based on the first and second image features, thus obtaining the loss value of the second adversarial loss function. Here, the model training device performs backpropagation in the discriminator based on this second adversarial loss function loss value to adjust the discriminator's model parameters, thereby training the discriminator. Furthermore, when the image augmentation model training is iterative, the discriminator training process is also iterative; and the discriminator's iteration proceeds along with the image augmentation model iteration and ends with the completion of the image augmentation model iteration.

[0199] In this embodiment, the model training device determines the loss value of the second adversarial loss function based on the second image features and the first image features. This includes: the model training device first determines the second type label of the enhanced sample image and then determines the third type label of the label image; finally, it calculates the loss value of the second adversarial loss function based on the second type label, the first image features, the third type label, and the second image features. The second adversarial loss function is used to train the discriminator, measuring its accuracy and providing direction for training; that is, the discriminator is trained by minimizing the loss value of the second adversarial loss function. Furthermore, the loss value of the second adversarial loss function is negatively correlated with the accuracy of the discriminator.

[0200] It should be noted that, since the discriminator is used to identify augmented sample images as fake images, the second type label of the augmented sample image is the fake label of the augmented sample image for the discriminator, for example, denoted as 0; while since the discriminator is used to identify labeled images as real images, the third type label of the labeled image is the real label of the augmented sample image for the discriminator, for example, denoted as 1. Here, the model training device calculates the loss value based on the third type label and the first image features, for example, by calculating cross-entropy loss, and the calculated loss value is called the first sub-adversarial loss value; and calculates the loss value based on the third type label and the second image features, for example, by calculating cross-entropy loss, and the calculated loss value is called the second sub-adversarial loss value; finally, the first sub-adversarial loss value and the second sub-adversarial loss value are combined into the loss value of the second adversarial loss function, which is used to characterize the mapping relationship between the first image features, the second image features and the loss value of the second adversarial loss function in the adversarial learning mode of the discriminator; wherein, the model training device can combine the first sub-adversarial loss value and the second sub-adversarial loss value in the following ways: accumulation, weighted summation, etc.

[0201] The image processing method provided in this application will be described below with reference to exemplary applications and implementations of the electronic device provided in the embodiments of this application. Furthermore, the image processing method provided in the embodiments of this application is applied to various image enhancement scenarios such as cloud technology, artificial intelligence, smart transportation, medical applications, and automotive applications.

[0202] See Figure 9A , Figure 9A This is a flowchart illustrating the image processing method provided in an embodiment of this application; wherein, Figure 9A The main body executing each step is the image processing device, which will be discussed below. Figure 9A The steps shown are explained.

[0203] Step 108: In response to the image enhancement request, obtain the image to be enhanced as requested in the image enhancement request.

[0204] It should be noted that when a request is made to enhance the image to be enhanced, the image processing device receives the image enhancement request. In response to this request, the image processing device can obtain the image to be enhanced from the image enhancement request, or it can obtain the image to be enhanced from the image repository based on the image identifier requested to be enhanced in the image enhancement request. This embodiment of the application does not limit this. The image enhancement request is a request for image enhancement, such as a request generated upon receiving a touch on an image enhancement control, a request generated upon receiving an instruction to enhance the image, etc.; the image to be enhanced is the image requested to be enhanced by the image enhancement request, i.e., the image to be enhanced.

[0205] Step 109: Enhance the image to be enhanced using the trained image enhancement model to obtain the enhanced image.

[0206] It should be noted that the trained image augmentation model can reside on the model training device, be deployed on the image processing device, or be deployed on a server other than the model training device and the image processing device. When the trained image augmentation model is located on the model training device, the image processing device sends the image to be augmented to the model training device, so that the model training device can augment the image using the trained image augmentation model and return the augmented image to the image processing device. When the trained image augmentation model is deployed on the image processing device, the image processing device augments the image using the deployed trained image augmentation model to obtain the augmented image. When the trained image augmentation model is deployed on a server other than the model training device and the image processing device, the image processing device sends the image to be augmented to the other server, so that the other server can augment the image using the deployed trained image augmentation model and return the augmented image to the image processing device. The trained image augmentation model is obtained through the training method of the image augmentation model provided in the embodiments of this application.

[0207] In the embodiments of this application, the model training device and the image processing device can be different electronic devices or the same electronic device, and the embodiments of this application do not limit this.

[0208] based on Figure 6B See Figure 9B , Figure 9B The embodiments provided in this application are related to Figure 9A The corresponding logical flow diagram; such as Figure 9BAs shown, when an image enhancement request 9-1 is received, the image to be enhanced, 9-2, is obtained in response to the request. The image to be enhanced, 9-2, is then input into the trained image enhancement model 6-7 for enhancement, resulting in the enhanced image 9-3.

[0209] It is understood that, since the training method of the image enhancement model provided in this application embodiment can improve the accuracy of the trained image enhancement model, the quality of the enhanced image can be improved when the trained image enhancement model is used to enhance the image to be enhanced. Because the semantic meaning of the labeled image is obtained based on the descriptive text of the labeled image, the trained image enhancement model can improve the consistency between image structure and image semantics; therefore, when the trained image enhancement model is used to enhance the image to be enhanced, the consistency between the image structure and image semantics of the enhanced image can be improved.

[0210] In this embodiment of the application, the image processing device enhances the image to be enhanced using a trained image enhancement model. After obtaining the enhanced image, the image processing method further includes: the image processing device displaying the enhanced image; or, determining the image classification result of the enhanced image and displaying the enhanced image and the image classification result.

[0211] It should be noted that image processing devices can directly display augmented images, and can also perform image processing on augmented images, such as image classification, and output the obtained image classification results and the augmented image. Furthermore, image classification can include entity classification, classification based on the level of image content, and so on.

[0212] It is understood that since the training method of the image enhancement model provided in this application embodiment can improve the accuracy of the trained image enhancement model, when the image to be enhanced is enhanced by the trained image enhancement model, the quality of the enhanced image can be improved, and thus the accuracy of the image classification result of the enhanced image can be improved.

[0213] The following describes an exemplary application of the embodiments of this application in a practical application scenario. This exemplary application describes the process of enhancing the illumination of an endoscopic image. It is readily apparent that the image processing method provided by the embodiments of this application is applicable to various image processing scenarios requiring illumination enhancement, such as endoscopic images, night scene images, and indoor images. Here, the process of enhancing the illumination of an endoscopic image is used as an example for illustration.

[0214] It's important to note that endoscopic imaging is a medical imaging technique used to examine internal organs and blood vessels. An endoscope is a flexible, tubular device, usually equipped with a camera, that can be inserted into the body through natural openings or incisions. It is commonly used for examining the digestive system, such as the esophagus, stomach, small intestine, and colon; the respiratory system, such as the trachea and bronchi; the cardiovascular system, such as cardiac endoscopy; and other areas, such as the abdomen, pelvis, and other organs. Due to the operation of endoscopic imaging, the environment of the organs being imaged, and the endoscopic equipment itself, the acquired endoscopic images often suffer from noise, poor visibility (below the visibility threshold), and low contrast (below the contrast threshold). These images are referred to as low-light endoscopic images. Low-light endoscopic images affect post-processing results such as classification and analysis; therefore, illumination enhancement is often performed on low-light endoscopic images. Among them, the brightness threshold, visibility threshold and contrast threshold can all be set manually or determined based on statistical data. This application embodiment does not limit this. The task of enhancing the illumination of low-light endoscopic images is called the Low-Light Endoscopic Image Enhancement (LLEIE) task.

[0215] See Figure 10 , Figure 10 This is a schematic diagram of an exemplary training image enhancement model provided in an embodiment of this application; as shown... Figure 10 As shown, the low-light endoscope image 10-11 (referred to as the sample image) to be enhanced is enhanced using the baseline model 10-21 (referred to as the image enhancement model) to obtain the enhanced image 10-12 (referred to as the enhanced sample image); for the label image 10-13 of the low-light endoscope image 10-11, the text modulation semantics 10-14 (referred to as the label image semantics) is obtained; the enhanced image 10-12, the label image 10-13, and the text modulation semantics 10-14 are passed through the text modulation semantics-aware discriminator 10-22 (referred to as the discriminator) to obtain the adversarial loss 10-31 (referred to as the loss value of the first adversarial loss function and the loss value of the second adversarial loss function); finally, the text modulation semantics-aware discriminator 10-22 and the baseline model 10-21 are trained using backpropagation based on the adversarial loss 10-31 to obtain the trained baseline model 10-21 (referred to as the trained image enhancement model).

[0216] The following describes the process of obtaining text modulation semantics 10-14.

[0217] It should be noted that the text modulation semantics 10-14 are obtained by modulating the label images 10-13 and the prompt text of the label images 10-13 through a text modulator, while the prompt text of the label images 10-13 is obtained through prompt learning.

[0218] See Figure 11 , Figure 11 This is the process of obtaining prompt text for the label image provided in the embodiments of this application; such as Figure 11 As shown, CLIP includes an image encoder 11-11 (denoted as Φ). img ()) and text encoder 11-12 (denoted as Φ) text ()); Image encoder 11-11 is used to encode the label image 10-13 and the low-light endoscope image 11-21 (referred to as auxiliary sample image) to be enhanced, respectively; text encoder 11-12 is used to encode the initial text prompt 11-31 (referred to as the first initialization text) and the initial text prompt 11-32 (referred to as the second initialization text), respectively; based on the encoding results of the label image 10-13 and the low-light endoscope image 11-21, and the encoding results of the initial text prompt 11-31 and the initial text prompt 11-32, the cross-entropy loss 11-4 (denoted as L) of the initial text prompt 11-31 and the initial text prompt 11-32 is calculated, respectively. bce , which is called the loss value of the second loss function); finally, with the parameters of the image encoder 11-11 and the text encoder 11-12 frozen, the initial text prompts 11-31 and 11-32 are adjusted by backpropagation based on the cross-entropy loss 11-4 until the optimization of the initial text prompts 11-31 is completed, thus obtaining the text prompts of the label image 10-13; it is easy to see that the text prompts of the label image 10-13 can represent the text description of the endoscope image under normal lighting conditions.

[0219] It should be noted that label images 10-13 and low-light endoscope images 11-21 may or may not correspond. When label images 10-13 and low-light endoscope images 11-21 correspond, then low-light endoscope image 11-21 is... Figure 10 The low-light endoscope image 10-11 is used. The initial text prompt 11-31 is obtained by initializing the text prompt for the label image 10-13, and the initial text prompt 11-32 is obtained by initializing the text prompt for the low-light endoscope image 11-21; the number of tokens for both the initial text prompt 11-31 and the initial text prompt 11-32 is N. The cross-entropy loss 11-4 can be obtained by formula (1) and formula (2), which are shown below.

[0220]

[0221] Among them, I n Represents labeled images 10-13, I l Images 11-21 are from a low-light endoscope; Φ img (I n) represents the encoding result of the label image 10-13, Φ img (i n ) represents the encoding result of low-light endoscopy images 11-21; T n This indicates the initial text prompt 11-31, T l This indicates the initial text prompt 11-32; Φ text (T n ) represents the encoding result of the initial text prompt 11-31, Φ text (T l ) represents the encoding result of the initial text prompt 11-32; e is the natural constant; cos(Φ img (I n ), Φ text (T n The similarity between the labeled images 10-13 and the initial text prompts 11-31 is called the first similarity, cos(Φ) img (I l ), Φ text (T l The similarity between the low-light endoscopic image 11-21 and the initial text prompt 11-32 is called the second similarity, cos(Φ) img (I l ), Φ text (T n )) indicates the similarity between low-light endoscopic images 11-21 and the initial text prompt 11-31; for or Indicate i n The predicted probability, Indicate i l The predicted probability; y∈{0,1}, for I n y is 1, while for I l y is 0.

[0222] See Figure 12 , Figure 12 This is an exemplary text modulation diagram provided in an embodiment of this application; as shown... Figure 12 As shown, the text prompt is 12-1 (denoted as...). The text prompt for the label images 10-13 (referred to as the first descriptive text) is used here. Figure 11 The text encoder 11-12 encodes the text prompt 12-1 to obtain the text embedding 12-2 (denoted as...). As shown in formula (3). Image feature extraction module 12-3 (denoted as Φ) is used. pvm ()) Perform feature extraction on label image 10-13 to obtain image feature 12-4 (denoted as As shown in formula (4).

[0223]

[0224] See also Figure 12 In order to improve The semantic richness and effectiveness, through coupling and We obtain text modulation semantics 10-14 (denoted as...) Here, coupling is achieved through text modulator 12-5 (denoted as TextModulator()). and In other words, using a multilayer perceptron (MLP) to... An affine transformation is performed to obtain the scaling parameter γ and the bias term β, as shown in formula (5). Since the scaling parameter γ is the dot product parameter and the bias term is the addition parameter, and in order to preserve the original semantic information... Obtained by residual connection As shown in formula (6);

[0225]

[0226] The following describes the processing procedure of the text modulation semantic perception discriminator 10-22.

[0227] It should be noted that the text modulation semantic awareness discriminator 10-22 includes at least one semantic guidance block, and each semantic guidance block includes a text modulator and a convolutional modulation module.

[0228] See Figure 13 , Figure 13 This is a schematic diagram of an exemplary text modulation semantic-aware discriminator provided in an embodiment of this application; as shown... Figure 13As shown, an exemplary text modulation semantic awareness discriminator 10-22 is illustrated, comprising three semantic guidance blocks (semantic guidance blocks 13-11 to 13-13). The input to semantic guidance block 13-11 is an endoscopy image 13-21 (denoted as F0), text modulation semantics 10-14, and text embedding 12-2, wherein the endoscopy image 13-21 can be the enhanced image 10-12 or the labeled image 10-13. Here, semantic guidance block 13-11 processes endoscopic image 13-21 using text modulation semantics 10-14 and text embedding 12-2 to obtain semantically perceived image feature 13-22 (denoted as F1, also known as intermediate image feature); semantic guidance block 13-12 processes semantically perceived image feature 13-22 using text modulation semantics 10-14 and text embedding 12-2 to obtain semantically perceived image feature 13-23 (denoted as F2, also known as intermediate image feature); semantic guidance block 13-11 processes semantically perceived image feature 13-23 using text modulation semantics 10-14 and text embedding 12-2 to obtain semantically perceived image feature 13-24 (denoted as F3, also known as first image feature or second image feature). The adversarial loss 10-31 is determined based on semantically perceived image feature 13-24.

[0229] The following explanation uses semantic guide blocks 13-11 as an example to illustrate semantic guide blocks.

[0230] See Figure 14 , Figure 14 This is a schematic diagram of an exemplary semantic guidance block provided in an embodiment of this application; as shown in 14, the semantic guidance block 13-11 includes convolutional blocks (ConvBlock) 14-11 (the corresponding processing process is denoted as ConvBlock()) for processing endoscopic images 13-21 and a 1*1 convolutional layer (Convolution, the corresponding processing process is denoted as Conv()). 1×1 ())14-12, through convolutional blocks 14-11 and convolutional layers 14-12, the endoscopic image 13-21 is processed sequentially to obtain image features 14-21 (denoted as F). img As shown in formula (7). The semantic guidance block 13-11 also includes a 1*1 convolutional layer 14-31, a group normalization layer (GroupNorm) 14-32, a layer normalization module (LayerNorm) 14-33, a convolutional modulation module 14-34, a text modulator 14-35, a layer normalization module 14-36, a cross attention layer (Cross Attention) 14-37, a layer normalization module 14-38, and an activation layer (GELU) 14-39; through the convolutional layer 14-31 (the corresponding processing is denoted as Conv 1×1The grouping normalization layer 14-32 (the corresponding processing step is denoted as GN()) and the layer normalization module 14-33 (the corresponding processing step is denoted as LN()) sequentially normalize the text modulation semantics 10-14 to obtain normalized features 14-41 (denoted as...). As shown in formula (8); the normalized features 14-41 are processed by the convolution modulation module 14-34, and the semantic features 14-42 (denoted as...) are... As shown in formulas (9) to (11); the semantic features 14-42 are processed sequentially by the text modulator 14-35 (the corresponding processing process is denoted as TextModulator()) and the layer normalization module 14-36 (the corresponding processing process is denoted as LN()) to obtain the query features 14-43(Q), as shown in formula (12); the key (K) / value (V) of the query features 14-34 and the image features 14-21 are processed by the cross attention layer 14-37 to obtain the attention features (denoted as F). att The attention features are processed sequentially by the layer normalization module 14-38 (the corresponding processing process is denoted as LN()) and the activation layer 14-39 (the corresponding processing process is denoted as GELU()). The processing results of the attention features and the image features 14-21 are channel concatenated (denoted as C, the corresponding processing process is denoted as Concat()) to obtain the semantic perception image features 13-22, as shown in formula (13).

[0231] F img =Conv 1×1 (ConvBlock(F0)) (7);

[0232]

[0233] The processing steps corresponding to the convolution modulation modules 14-34 include 1*1 convolutional layer processing (Conv). 1×1 ( ), GELU activation layer processing, DConv M*M convolutional layer processing M×M () and dot product processing ⊙; A represents the weight of normalized feature 14-41 (called image semantic weight), and V1 is called image semantic feature.

[0234]

[0235] F1 = Concat(GELU(LN(F)) att )), F img (13);

[0236] It should be noted that the resistance loss 10-31 is as shown in formula (14).

[0237]

[0238] Where, min G max D L gan Indicates the resistance loss L gan This enables the text modulation semantic awareness discriminator 10-22 to distinguish between the enhanced image 10-12 and the label image 10-13, and enables the baseline model 10-21 to generate an enhanced image similar to the label image 10-13. This represents the probability distribution of the labeled image 10-13. Let represent the probability distribution of low-light endoscopic images 10-11; D represents the text modulation semantic perception discriminator; G represents the baseline model used to enhance low-light endoscopic images; and E represents the expectation.

[0239] It should be noted that the loss L of the baseline model G As shown in formula (15).

[0240] L G =L b +γ p L p +γ gan L gan (15);

[0241] Among them, L b L represents the initial loss of the baseline model itself (called the model loss value). p L represents the perceived loss (called the perceived loss value). gan γ represents the adversarial loss of the baseline model (the loss value referred to as the first adversarial loss function); p and γ gan This represents the weighting coefficient, for example, γ. p γ is 0.01. gan It is 0.1.

[0242] See Figure 15 , Figure 15 This is an exemplary model training flowchart provided in an embodiment of this application; as shown... Figure 15 As shown, the execution entity of this exemplary model training process is a server, and the exemplary model training process includes steps 201 to 206. Each step will be described below.

[0243] Step 201: Start training the model.

[0244] It should be noted that training the model begins when the baseline model is selected.

[0245] Step 202: Iteratively train the discriminator and baseline model based on the training dataset and training settings.

[0246] It should be noted that the discriminator can be a Text-Modulated Semantic-Aware Discriminator (TMSD). Here, we first obtain text cues representing a normal lighting scene using CLIP, and use the following training settings: the number of tokens involved in the text cues is 16, and the learning rate is 5×10. -6 The iteration count was 10,000, the batch size was 8, and all images were resized to 512×512 pixels. Then, the text modulation semantics of the normal lighting images obtained from text cues in normal lighting scenes were injected into the patch discriminator, which was then trained using an optimizer. When the baseline model was LLCaps, the initial learning rate of TMSD was 1×10⁻⁶. -6 When the baseline model is another baseline model, the initial learning rate of TMSD is 1×10. -4 .

[0247] It should be noted that TMSD is a technique used to improve the performance of Generative Adversarial Networks (GANs). In image generation tasks, it is used to enhance the semantic clarity and texture realism of generated images. In the embodiments of this application, text and images can be combined using discriminators such as TMSD. By modulating the text, the semantic information in the image can be enhanced, enabling the generator (called the image enhancement model) to not only generate visually realistic images but also ensure that the image is semantically consistent with the given text.

[0248] Step 203: Update the parameters of the discriminator and the baseline model based on the training objective function and optimizer.

[0249] It should be noted that backpropagation is performed between the discriminator and the baseline model based on the training objective function and optimizer to update the parameters of the discriminator and the baseline model.

[0250] Step 204: Determine whether the set number of iterations has been reached.

[0251] It should be noted that the training of the discriminator and the baseline model includes a pre-set number of iterations; if so, proceed to step 205; otherwise, proceed to step 202. The pre-set number of iterations, for example, is 1000, which can be manually set or determined based on statistical data; this embodiment does not limit this.

[0252] Step 205: Obtain the trained baseline model.

[0253] It should be noted that the baseline model that completes the set number of iterations is the trained baseline model.

[0254] Step 206: End model training.

[0255] For example, see Figure 16 , Figure 16 This is an exemplary model testing flowchart provided in an embodiment of this application; as shown... Figure 16 As shown, the execution entity of this exemplary model testing process is the server, and the exemplary model testing process includes steps 301 to 304. Each step will be described below.

[0256] Step 301: Obtain the endoscope image to be tested.

[0257] It should be noted that the endoscopic image to be tested is the same as the endoscopic image to be enhanced.

[0258] Step 302: Load the trained baseline model.

[0259] It should be noted that the pre-trained baseline model loaded here is... Figure 10 The pre-trained baseline model.

[0260] Step 303: Enhance the endoscope image to be tested using the loaded, trained baseline model to obtain the enhanced endoscope image.

[0261] It should be noted that the enhancement of the endoscope image to be tested is achieved by performing forward computation on the loaded, trained baseline model, so as to obtain the enhanced endoscope image.

[0262] Step 304: Output the enhanced endoscopic image.

[0263] It should be noted that after obtaining the enhanced endoscopic image, the enhanced endoscopic image can be output by displaying the enhanced endoscopic image, or by sending the enhanced endoscopic image to the terminal, etc. The embodiments of this application do not limit this.

[0264] See Figure 17 , Figure 17 This is an exemplary image enhancement diagram provided in an embodiment of this application; as shown... Figure 17As shown, the front end 17-11 (referred to as the image processing device) sends the endoscope image 17-21 to be enhanced to the back end 17-12 (referred to as the model training device). The trained image enhancement model 17-3 of the back end 17-12 enhances the endoscope image 17-21 to be enhanced, resulting in the enhanced endoscope image 17-22. The front end 17-13 outputs the enhanced endoscope image 17-22. The front end 17-11 and the front end 17-13 may be the same or different, and this embodiment does not limit this.

[0265] See Figure 18 , Figure 18 This is another exemplary image enhancement diagram provided in the embodiments of this application; as shown Figure 18 As shown, the trained image enhancement model is used to enhance the endoscope image 18-1 (referred to as the image to be enhanced) to obtain the enhanced endoscope image 18-2 (referred to as the enhanced image).

[0266] In this embodiment, the effectiveness of the image augmentation model trained in this embodiment was evaluated using five public datasets. These five public datasets include Dataset 1 (synthetic dataset KC), Dataset 2 (synthetic dataset RLE), Dataset 3 (synthetic dataset Endo4IE_UE), Dataset 4 (real-world clinical dataset EAD2020), and Dataset 5 (clinical downstream segmentation dataset (RLE-Seg)). Based on these five public datasets, it was demonstrated that integrating TMSD into the baseline model can improve the perceptual quality of the augmented image and enhance image segmentation accuracy. See Tables 1, 2, and 3, which describe the results of evaluating the five baseline models and integrating TMSD into them on the five public datasets using various evaluation metrics.

[0267] Table 1

[0268]

[0269]

[0270] Table 2

[0271]

[0272]

[0273] Table 3

[0274]

[0275] As shown in Tables 1, 2 and 3, the baseline model integrating TMSD has better image enhancement performance than the baseline model under various evaluation metrics.

[0276] It is understood that the embodiments of this application, through cue learning, mine the multimodal priors of CLIP to obtain text cues corresponding to normal light endoscope images to represent normal lighting scenes; wherein, the normal light endoscope image is described relative to the low-light endoscope image, indicating that the illumination intensity of the image acquisition environment of the endoscope image is greater than a specified illumination intensity. Furthermore, by using a text modulator to coordinate text cues and normal light endoscope images, the richness of the semantic representation of normal light endoscope images can be improved. Moreover, by integrating a text-modulated semantic perception discriminator into the training process of the baseline model, and employing convolutional modulation and cross-attention mechanisms, the semantic guidance information obtained by combining text cues and normal light endoscope images is fused into the text-modulated semantic perception discriminator. This promotes fine-grained distribution learning of visual semantics in normal light endoscope images, guides different enhancement baselines for image enhancement, ensures that the enhanced structure is consistent with the semantic information, improves the visual quality of the enhanced image, and reduces the impact on the inference process of the baseline model, thereby improving the accuracy of subsequent diagnostic analysis.

[0277] The following description continues to illustrate the exemplary structure of the model training device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 4 As shown, the software modules stored in the model training device 455 in the memory 450 may include:

[0278] The model enhancement module 4551 is used to acquire a sample image and a label image, wherein the label image is a reference enhancement result of the sample image; and to enhance the sample image using an image enhancement model to obtain an enhanced sample image.

[0279] Text determination module 4552 is used to determine first descriptive text, which is used to describe the image content of the label image;

[0280] The semantic extraction module 4553 is used to extract the semantic meaning of the label image based on the first descriptive text;

[0281] The loss determination module 4554 is used to extract the first image features of the enhanced sample image based on the semantics of the labeled image; and to determine the loss value of the target loss function based on the first image features and the target loss function preset for the image enhancement model.

[0282] The model training module 4555 is used to train the image enhancement model based on the loss value of the target loss function to obtain the trained image enhancement model.

[0283] In this embodiment of the application, the target loss function includes a first adversarial loss function. The loss determination module 4554 is further configured to determine the loss value of the first adversarial loss function based on the first image features and the first adversarial loss function. The first adversarial loss function is used to characterize the mapping relationship between the first image features and the loss value of the first adversarial loss function in the adversarial learning mode of the image enhancement model. The loss value of the target loss function is determined based on the loss value of the first adversarial loss function.

[0284] In this embodiment of the application, the loss determination module 4554 is further configured to determine a target query feature based on a first text feature and the semantics of the labeled image, wherein the first text feature is a feature of the first descriptive text and the target query feature is an information feature to be queried in the enhanced sample image; determine a first key feature and a first value feature of the enhanced sample image, wherein the first key feature is an information feature in the enhanced sample image to be compared with the target query feature and the first value feature is a content feature in the enhanced sample image corresponding to the first key feature; and perform attention processing on the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image.

[0285] In this embodiment of the application, the loss determination module 4554 is further configured to enhance the semantics of the labeled image to obtain the semantics of the target image; and to fuse the first text features and the semantics of the target image to obtain the target query features.

[0286] In this embodiment of the application, the loss determination module 4554 is further configured to determine the image semantic weights of the labeled image semantics, the image semantic weights being used to enhance the labeled image semantics; to perform convolution on the labeled image semantics to obtain image semantic features; and to fuse the image semantic weights and the image semantic features to obtain the target image semantics.

[0287] In this embodiment, the loss determination module 4554 is further configured to perform attention processing on the target query feature, the first key feature, and the first value feature to obtain intermediate image features of the enhanced sample image; determine the second key feature and the second value feature of the intermediate image features, wherein the second key feature is an information feature in the intermediate image features to be compared with the target query feature, and the second value feature is a content feature in the intermediate image features corresponding to the second key feature; and perform attention processing on the target query feature, the second key feature, and the second value feature to obtain the first image feature of the enhanced sample image.

[0288] In this embodiment of the application, the target loss function further includes a first loss function. The loss determination module 4554 is further configured to determine the loss value of the first loss function based on the enhanced sample image, the label image, and the first loss function. The loss value of the first loss function includes at least one of a model loss value and a perceptual loss value. The model loss value is calculated using the loss function corresponding to the image enhancement model, and the perceptual loss value is obtained by calculating the difference between the enhanced sample image and the label image in the feature space. Based on the loss value of the first loss function and the loss value of the first adversarial loss function, the loss value of the target loss function is determined.

[0289] In this embodiment of the application, the loss determination module 4554 is further configured to determine a first type label of the enhanced sample image, wherein the first type label is the true label of the enhanced sample image; and determine the loss value of the first adversarial loss function based on the first type label, the first image features, and the first adversarial loss function.

[0290] In this embodiment, the extraction of the first image feature of the enhanced sample image based on the semantics of the labeled image is implemented by a discriminator. The model training module 4555 is further configured to extract the second image feature of the labeled image based on the semantics of the labeled image using the discriminator; determine the loss value of a second adversarial loss function based on the second image feature and the first image feature, wherein the second adversarial loss function is used to characterize the mapping relationship between the second image feature and the first image feature and the loss value of the second adversarial loss function in the adversarial learning mode; and train the discriminator based on the loss value of the second adversarial loss function to obtain an intermediate discriminator.

[0291] In this embodiment of the application, the model training module 4555 is further configured to train the image enhancement model based on the loss value of the target loss function to obtain an intermediate image enhancement model; and to train the intermediate image enhancement model through the intermediate discriminator to obtain the trained image enhancement model.

[0292] In this embodiment of the application, the model training module 4555 is further configured to determine a second type label of the augmented sample image, wherein the second type label is a false label of the augmented sample image; determine a third type label of the label image, wherein the third type label is a true label of the label image; and calculate a loss value based on the second type label and the first image feature, and the third type label and the second image feature, to obtain the loss value of the second adversarial loss function.

[0293] In this embodiment of the application, the semantic extraction module 4553 is further configured to extract the first text features of the first descriptive text; extract the label image features of the label image; and fuse the first text features and the label image features to obtain the label image semantics.

[0294] In this embodiment of the application, the semantic extraction module 4553 is further configured to perform an affine transformation on the first text features to obtain the scaling parameters and bias terms of the label image features; fuse the scaling parameters with the label image features to obtain a feature fusion result; and combine the label image features, the bias terms, and the feature fusion result into the label image semantics.

[0295] In this embodiment of the application, the text determination module 4552 is further configured to determine a first initialization text of the label image and a second initialization text of an auxiliary sample image, wherein the auxiliary sample image is used to assist the label image in optimizing the first initialization text; determine a first similarity between the label image and the first initialization text, and a second similarity between the auxiliary sample image and the second initialization text; determine a loss value of a second loss function between the first initialization text and the label image based on the first similarity and the second similarity; and adjust the first initialization text based on the loss value of the second loss function to obtain the first descriptive text.

[0296] The following description continues to illustrate the exemplary structure of the model training device 456 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 5 As shown, the software modules stored in the model training device 456 in the memory 450 may include:

[0297] Image acquisition module 4561 is used to obtain the image to be enhanced as requested by the image enhancement request in response to the image enhancement request;

[0298] The model application module 4562 is used to enhance the image to be enhanced by the trained image enhancement model to obtain an enhanced image. The trained image enhancement model is obtained by the training method of the image enhancement model provided in the embodiments of this application.

[0299] In this embodiment of the application, the model training device 456 further includes a result output module 4563, used to display the enhanced image; or, to determine the image classification result of the enhanced image and display the enhanced image and the image classification result.

[0300] This application provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the image enhancement model training method or image processing method described in this application embodiment.

[0301] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training method of the image enhancement model provided in this application. For example, ... Figure 6A The training method of the image enhancement model is shown; or, it causes the processor to execute the image processing method provided in the embodiments of this application, for example, such as... Figure 9A The image processing method shown.

[0302] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0303] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0304] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0305] As an example, computer-executable instructions can be deployed to execute on a single electronic device (in this case, the single electronic device is a model training device and an image processing device), or to execute on multiple electronic devices located in one location (in this case, the multiple electronic devices located in one location are a model training device and an image processing device), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in this case, the multiple electronic devices distributed across multiple locations and interconnected via a communication network are a model training device and an image processing device).

[0306] It is understood that in the embodiments of this application, data such as images are involved. When the embodiments of this application are applied to specific products or technologies, the permission or consent of the information subject is required. The collection, use and processing of related data must comply with relevant laws, regulations and standards, and conform to the principles of legality, legitimacy and necessity. It does not involve obtaining data types prohibited or restricted by laws and regulations, and will not hinder the normal operation of the target website.

[0307] In summary, after obtaining the enhanced sample image of the sample image through the image enhancement model, the embodiments of this application extract the label image semantics of the label image based on the first descriptive text corresponding to the label image of the sample image, extract the first image features of the enhanced sample image based on the label image semantics, and determine the loss value for training the image enhancement model based on the first image features and the target loss function pre-set for the image enhancement model, so as to achieve the training of the image enhancement model. In the training process of the image enhancement model, the first image feature used to determine the loss value of the target loss function is extracted based on the semantics of the label image. This semantics is extracted from the label image based on the first descriptive text. The first descriptive text represents the textual dimension of the label image, which differs from the label image's information in another dimension. Therefore, extracting the semantics of the label image based on the first descriptive text is a process of determining the semantics of the label image through information from different dimensions. This improves the comprehensiveness of the label image semantics in terms of feature dimensions, thus enhancing its accuracy in representing the reference enhancement result. Consequently, extracting the first image feature used to calculate the loss value of the target loss function based on the semantics of the label image improves the accuracy of the target loss function's loss value. Furthermore, training the image enhancement model based on this target loss function's loss value improves the accuracy of the trained image enhancement model. Additionally, when the trained image enhancement model is used to enhance the image to be enhanced, it improves the quality of the enhanced image, thereby improving the accuracy of the image classification results for the enhanced image.

[0308] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for an image enhancement model, characterized in that, The method includes: Obtain a sample image and a label image, wherein the label image is a reference enhancement result of the sample image; The sample image is enhanced using an image enhancement model to obtain an enhanced sample image; Determine a first descriptive text, which is used to describe the image content of the label image; Extract the semantic meaning of the label image from the label image based on the first descriptive text; Based on the semantics of the labeled image, the first image feature of the enhanced sample image is extracted; Based on the first image features and the target loss function pre-set for the image enhancement model, the loss value of the target loss function is determined; The image enhancement model is trained based on the loss value of the target loss function to obtain the trained image enhancement model.

2. The method according to claim 1, characterized in that, The target loss function includes a first adversarial loss function. Determining the loss value of the target loss function based on the first image features and a pre-defined target loss function for the image enhancement model includes: Based on the first image features and the first adversarial loss function, the loss value of the first adversarial loss function is determined. The first adversarial loss function is used to characterize the mapping relationship between the first image features and the loss value of the first adversarial loss function in the adversarial learning mode of the image enhancement model. The loss value of the target loss function is determined based on the loss value of the first adversarial loss function.

3. The method according to claim 1 or 2, characterized in that, The step of extracting the first image feature of the enhanced sample image based on the semantics of the labeled image includes: Based on the first text feature and the semantics of the labeled image, the target query feature is determined, wherein the first text feature is the feature of the first descriptive text, and the target query feature is the information feature to be queried in the enhanced sample image; Determine the first key feature and the first value feature of the enhanced sample image, wherein the first key feature is the information feature in the enhanced sample image to be compared with the target query feature, and the first value feature is the content feature in the enhanced sample image corresponding to the first key feature; Attention processing is performed on the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image.

4. The method according to claim 3, characterized in that, The determination of target query features based on the first text features and the semantics of the label image includes: The semantics of the labeled image are enhanced to obtain the semantics of the target image; The target query features are obtained by fusing the first text features and the target image semantics.

5. The method according to claim 4, characterized in that, The step of enhancing the semantics of the labeled image to obtain the semantics of the target image includes: Determine the image semantic weights of the labeled image, and use the image semantic weights to enhance the semantics of the labeled image; The semantic features of the labeled image are obtained by convolving the semantic features of the image. The image semantic weights and image semantic features are fused to obtain the target image semantics.

6. The method according to any one of claims 3 to 5, characterized in that, The attention processing of the target query feature, the first key feature, and the first value feature to obtain the first image feature of the enhanced sample image includes: Attention processing is performed on the target query features, the first key features, and the first value features to obtain the intermediate image features of the enhanced sample image; Determine the second key feature and the second value feature of the intermediate image features, wherein the second key feature is the information feature in the intermediate image features to be compared with the target query feature, and the second value feature is the content feature in the intermediate image features corresponding to the second key feature; Attention processing is performed on the target query feature, the second key feature, and the second value feature to obtain the first image feature of the enhanced sample image.

7. The method according to claim 2, characterized in that, The target loss function further includes a first loss function, and determining the loss value of the target loss function based on the loss value of the first adversarial loss function includes: Based on the enhanced sample image, the label image, and the first loss function, the loss value of the first loss function is determined. The loss value of the first loss function includes at least one of a model loss value and a perceptual loss value. The model loss value is calculated using the loss function corresponding to the image enhancement model, and the perceptual loss value is obtained by calculating the difference between the enhanced sample image and the label image in the feature space. The loss value of the target loss function is determined based on the loss value of the first loss function and the loss value of the first adversarial loss function.

8. The method according to claim 2 or 7, characterized in that, The step of determining the loss value of the first adversarial loss function based on the first image features and the first adversarial loss function includes: Determine a first type label for the augmented sample image, wherein the first type label is the true label of the augmented sample image; Based on the first type label, the first image features, and the first adversarial loss function, the loss value of the first adversarial loss function is determined.

9. The method according to any one of claims 1 to 8, characterized in that, The extraction of the first image feature of the enhanced sample image based on the semantics of the labeled image is achieved through a discriminator. The method further includes: The discriminator extracts second image features of the label image based on the semantics of the label image. Based on the second image features and the first image features, the loss value of the second adversarial loss function is determined. The second adversarial loss function is used to characterize the mapping relationship between the second image features and the first image features and the loss value of the second adversarial loss function in the adversarial learning mode of the discriminator. The discriminator is trained based on the loss value of the second adversarial loss function to obtain an intermediate discriminator; The process of training the image enhancement model based on the loss value of the target loss function to obtain the trained image enhancement model includes: The image enhancement model is trained based on the loss value of the target loss function to obtain an intermediate image enhancement model; The intermediate image enhancement model is trained using the intermediate discriminator to obtain the trained image enhancement model.

10. The method according to claim 9, characterized in that, The step of determining the loss value of the second adversarial loss function based on the second image features and the first image features includes: Determine a second type of label for the augmented sample image, wherein the second type of label is a false label for the augmented sample image; Determine a third type of label for the label image, wherein the third type of label is the true label of the label image; The loss value of the second adversarial loss function is obtained by calculating the loss value based on the second type of label and the first image feature, and the third type of label and the second image feature.

11. The method according to any one of claims 1 to 10, characterized in that, The step of extracting the label image semantics from the label image based on the first descriptive text includes: Extract the first text feature from the first descriptive text; Extract the label image features from the label image; The first text features and the label image features are fused to obtain the label image semantics.

12. The method according to claim 11, characterized in that, The process of fusing the first text features and the label image features to obtain the label image semantics includes: Perform an affine transformation on the first text feature to obtain the scaling parameters and bias terms of the label image feature; The scaling parameters and the label image features are fused to obtain the feature fusion result; The label image features, the bias term, and the feature fusion result are combined to form the label image semantics.

13. The method according to any one of claims 1 to 12, characterized in that, The determination of the first description text includes: Determine a first initialization text for the label image and a second initialization text for an auxiliary sample image, wherein the auxiliary sample image is used to assist the label image in optimizing the first initialization text; Determine the first similarity between the label image and the first initialized text, and the second similarity between the auxiliary sample image and the second initialized text; Based on the first similarity and the second similarity, the loss value of the second loss function between the first initialization text and the label image is determined; The first initialization text is adjusted based on the loss value of the second loss function to obtain the first description text.

14. An image processing method, characterized in that, The method includes: In response to an image enhancement request, the image to be enhanced, which is requested to be enhanced, is obtained; The image to be enhanced is enhanced by the trained image enhancement model to obtain an enhanced image. The trained image enhancement model is obtained by the training method of the image enhancement model according to any one of claims 1 to 13.

15. The method according to claim 14, characterized in that, After enhancing the image to be enhanced using the trained image enhancement model to obtain the enhanced image, the method further includes: Display the enhanced image; Alternatively, determine the image classification result of the enhanced image and display the enhanced image and the image classification result.

16. A training device for an image enhancement model, characterized in that, The training device includes: The model enhancement module is used to acquire sample images and label images, wherein the label image is a reference enhancement result of the sample image; and to enhance the sample image using an image enhancement model to obtain an enhanced sample image. A text determination module is used to determine a first descriptive text, which is used to describe the image content of the label image; The semantic extraction module is used to extract the semantic meaning of the label image based on the first descriptive text; The loss determination module is used to extract the first image features of the enhanced sample image based on the semantics of the labeled image; and to determine the loss value of the target loss function based on the first image features and the target loss function pre-set for the image enhancement model. The model training module is used to train the image enhancement model based on the loss value of the target loss function, so as to obtain the trained image enhancement model.

17. An image processing apparatus, characterized in that, The image processing device includes: An image acquisition module is used to obtain the image to be enhanced as requested in the image enhancement request in response to the image enhancement request; The model application module is used to enhance the image to be enhanced using a trained image enhancement model to obtain an enhanced image. The trained image enhancement model is obtained by the training method of the image enhancement model according to any one of claims 1 to 13.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method of the image enhancement model according to any one of claims 1 to 13, or implements the image processing method according to claim 14 or 15.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training method of the image enhancement model according to any one of claims 1 to 13, or the image processing method according to claim 14 or 15.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the training method of the image enhancement model according to any one of claims 1 to 13, or the image processing method according to claim 14 or 15.

Images