Image processing model training method and device and image processing method

Abstract

The invention relates to an image processing model training method and device and an image processing method, and the method comprises the steps: obtaining training data which comprises a first endoscope image with an image degradation phenomenon, a text instruction corresponding to the first endoscope image, and a second endoscope image without the image degradation phenomenon; inputting the training data into an image processing model; the image processing model is used for converting the text instruction into a text embedding vector; extracting a semantic feature map of the first endoscope image based on an encoder; performing channel multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map; restoring the spatial resolution of the cross-modal feature map to the spatial resolution of the first endoscope image based on a decoder; superposing the cross-modal feature map and the semantic feature map, and outputting a superposed image; and adjusting parameters of the image processing model by taking the difference between the minimization model output image and the second endoscope image as a target. By adopting the method, the problem of degradation of various endoscope pictures can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer intelligent image processing, in particular to an image processing model training method and device and an image processing method. BACKGROUND

[0002] Endoscopy technology plays a crucial role in clinical medical diagnosis, especially in the early detection and real-time monitoring of diseases in the gastrointestinal tract, respiratory tract and other cavities. The high-definition image quality directly determines the accuracy of diagnosis and the safety of treatment. However, endoscopic images are often affected by various interference factors during the acquisition process, resulting in a significant decline in image quality. These disturbances include, but are not limited to, smoke generated by surgical operations (such as electrocautery smoke in polyp resection), lens stains (blood, tissue fluid, etc.), uneven lens lighting causing reflections or light spots, image noise, and motion blur caused by lens movement. These interference factors not only reduce the visual effect of the image, but also easily cause misdiagnosis or missed diagnosis, affecting the reliability of clinical decision-making.

[0003] In recent years, with the development of artificial intelligence technology, deep learning-based methods have made significant progress in the field of image processing. However, most related deep learning-based endoscopic image quality improvement methods only address one type of image degradation, such as improving lens stains, but not other issues such as uneven lighting, noise interference, and motion blur. In actual clinical scenarios, endoscopic images are often affected by multiple factors, and a single processing method cannot meet the complex and varied image quality improvement needs.

[0004] To address the problem of being unable to handle multiple endoscopic image degradation types in related technologies, there is currently no effective solution. SUMMARY

[0005] Therefore, it is necessary to provide an image processing model training method, device and image processing method that can handle multiple endoscopic image degradation types to address the above technical problems.

[0006] In a first aspect, an image processing model training method is provided in the present embodiment, the method comprising:

[0007] obtaining training data, the training data including a first endoscopic image with image degradation phenomena, a text instruction corresponding to the image degradation phenomena of the first endoscopic image, and a second endoscopic image without image degradation phenomena;

[0008] inputting the training data to an image processing model; wherein the image processing model is configured to convert the text instruction into a text embedding vector; extracting a semantic feature map of the first endoscopic image based on an encoder; performing channel-wise multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map; restoring a spatial resolution of the cross-modal feature map to a spatial resolution of the first endoscopic image based on a decoder symmetrically arranged with the encoder; superimposing the semantic feature map and the cross-modal feature map with restored spatial resolution to output a superimposed image;

[0009] adjusting parameters of the image processing model to minimize a difference between the image output by the image processing model and the second endoscopic image.

[0010] In some embodiments, adjusting parameters of the model to minimize a difference between the image output by the image processing model and the second endoscopic image comprises:

[0011] inputting the text embedding vector into a classifier to generate a first loss value based on a difference between an image degradation type predicted by the classifier and an image degradation type corresponding to the text embedding vector;

[0012] generating a second loss value based on a difference between the image output by the image processing model and the corresponding second endoscopic image;

[0013] adjusting parameters of the model based on the first loss value and the second loss value.

[0014] In some embodiments, generating a second loss value based on a difference between the image output by the image processing model and the second endoscopic image comprises:

[0015] comparing each pixel in the image output by the image processing model with a pixel at a corresponding position in the second endoscopic image to obtain a difference pixel with different pixel values, the difference pixel comprising a first pixel in the image output by the image processing model and a second pixel in the second endoscopic image;

[0016] generating the second loss value based on a mean square error between a pixel value of the first pixel and a pixel value of the corresponding second pixel.

[0017] In some embodiments, performing channel-wise multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map comprises:

[0018] multiplying the semantic feature map and the text embedding vector to obtain a first feature map;

[0019] assigning weights to the first feature map, such that elements in the first feature map associated with the image degradation phenomenon have higher weights than elements not associated with the image degradation phenomenon;

[0020] superimposing the first feature map with the assigned weights and the semantic feature map;

[0021] performing convolution processing on the superimposed feature maps to obtain the cross-modal feature map.

[0022] In some embodiments, converting the text instruction into a text embedding vector includes:

[0023] obtaining a text instruction corresponding to an image degradation phenomenon of a first endoscopic image;

[0024] converting the text instruction into a text embedding vector of a preset dimension based on a text encoder;

[0025] mapping the text embedding vector based on a linear transformation layer, such that a dimension of the mapped text embedding vector corresponds to the semantic feature map.

[0026] In some embodiments, extracting a semantic feature map of the first endoscopic image based on an encoder includes:

[0027] performing convolution processing on the first endoscopic image to obtain a second feature map;

[0028] assigning weights to the second feature map, such that elements in the second feature map associated with the image degradation phenomenon have higher weights than elements not associated with the image degradation phenomenon;

[0029] performing down-sampling on the second feature map with the assigned weights;

[0030] assigning weights to the down-sampled second feature map, such that elements in the down-sampled second feature map associated with the image degradation phenomenon have higher weights than elements not associated with the image degradation phenomenon, and obtaining the semantic feature map.

[0031] In some embodiments, obtaining training data includes:

[0032] simulating an image degradation phenomenon in an endoscopic image without the image degradation phenomenon to obtain a simulated endoscopic image;

[0033] integrating real endoscopic images with the image degradation phenomenon and the simulated endoscopic image to obtain a first endoscopic image in the training data.

[0034] In a second aspect, an image processing method is provided in the embodiments. The image processing method is applied to an image processing model, which is trained based on the image processing model training method of the first aspect. The image processing method comprises:

[0035] obtaining a target endoscopic image with image degradation phenomenon and a text instruction corresponding to the target endoscopic image;

[0036] inputting the target endoscopic image and the text instruction into the image processing model to obtain a first restored image.

[0037] In some embodiments, after obtaining the first restored image, the image processing method further comprises:

[0038] determining whether the first restored image contains image degradation phenomenon;

[0039] If yes, updating the text instruction based on the first restored image, and inputting the updated text instruction and the first restored image into the model to obtain a second restored image.

[0040] In a third aspect, an image processing model training device is provided in the embodiments. The device comprises:

[0041] a training data acquisition module configured to acquire training data, the training data comprising a first endoscopic image with image degradation phenomenon, a text instruction corresponding to the image degradation phenomenon of the first endoscopic image, and a second endoscopic image without image degradation phenomenon;

[0042] an input module configured to input the training data into an image processing model, wherein the image processing model is configured to convert the text instruction into a text embedding vector, extract a semantic feature map of the first endoscopic image based on an encoder, perform channel multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map, restore the spatial resolution of the cross-modal feature map to the spatial resolution of the first endoscopic image based on a decoder symmetrically arranged with the encoder, and superimpose the cross-modal feature map with restored spatial resolution and the semantic feature map to output a superimposed image.

[0043] a training module configured to adjust parameters of the image processing model to minimize the difference between an image output by the image processing model and the second endoscopic image.

[0044] The image processing model training method, device and image processing method train the image processing model through training data including the endoscope image and the text instruction, recognize and process different image degradation categories through the multi-modal fusion manner, and generate high-quality endoscope images, thereby solving the problem that multiple endoscope picture degradation types cannot be processed in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 An application environment diagram of the image processing model training method in an embodiment is shown in FIG. 1;

[0046] Figure 2 A flowchart of the image processing model training method in an embodiment is shown in FIG. 2;

[0047] Figure 3 A schematic diagram of the image processing model in an embodiment is shown in FIG. 3;

[0048] Figure 4 A schematic diagram of the attention module in an embodiment is shown in FIG. 4;

[0049] Figure 5 A schematic diagram of the instruction condition module in an embodiment is shown in FIG. 5;

[0050] Figure 6 A flowchart of the image processing method in an embodiment is shown in FIG. 6;

[0051] Figure 7 A structural block diagram of the image processing model training device in an embodiment is shown in FIG. 7;

[0052] Figure 8 An internal structure diagram of the computer device in an embodiment is shown in FIG. 8. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0054] The image processing model training method provided in the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 for processing. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The training data and pre-built image processing model are obtained from the data storage system, and the image processing model is trained through the training data. Among them, the terminal 102 can be, but not limited to, various personal computers, notebook computers, tablet computers. The server 104 can be realized by an independent server or a server cluster composed of multiple servers.

[0055] Figure 2 An image processing model training method is provided, and in the embodiment, as shown in the figure, the method comprises the following steps: Figure 2

[0056] Step 202, obtaining training data, the training data comprising a first endoscope image with image degradation phenomenon, a text instruction corresponding to the image degradation phenomenon of the first endoscope image, and a second endoscope image without image degradation phenomenon.

[0057] Among them, the first endoscope image includes a real endoscope image actually collected and existing image degradation phenomenon, and / or an endoscope image obtained by artificially simulating image degradation phenomenon on the basis of the second endoscope image. Different first endoscope images can correspond to different image degradation phenomena, and each first endoscope image can exist one or more image degradation phenomena. The second endoscope image corresponds to the first endoscope image. Optionally, the first endoscope image of different degradation types and its corresponding clean image (second endoscope image) are used as an image training data pair.

[0058] The text instruction includes the description of one or more image degradation phenomena contained in the first endoscope image. Optionally, various image degradation types involved in the acquisition of the first endoscope image are generated by a large language model, and the text description associated with various image degradation types is generated. Each image degradation type corresponds to multiple text descriptions. The text description generated by the large language model is used as the text instruction.

[0059] Step 204, inputting the training data into the image processing model; wherein the image processing model is used to convert the text instruction into a text embedding vector; extracting the semantic feature map of the first endoscope image based on the encoder; performing channel multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map; restoring the spatial resolution of the cross-modal feature map to the spatial resolution of the first endoscope image based on the decoder symmetrically arranged with the encoder; superimposing the semantic feature map and the cross-modal feature map with restored spatial resolution, and outputting the superimposed image.

[0060] ​The semantic feature map of the first endoscope image extracted based on the encoder has the same spatial dimension as the converted text embedding vector. The channel multiplication is a multiplication operation of corresponding pixels of the semantic feature map and the text embedding vector in the channel dimension. The encoder and the decoder are symmetrically arranged, that is, the two are symmetrical in network depth, layer structure, and feature map size change. For example, the encoder performs downsampling once, and the decoder corresponds to upsampling once.

[0061] Optionally, the first endoscope image and the text instruction corresponding to the same image degradation type are taken as a group, and multiple groups of training data corresponding to different image degradation types are input into the image processing model in batches.

[0062] Optionally, the encoder pre-processes the first endoscope image through a convolution layer, extracts semantic features of the first endoscope image, and then the semantic features sequentially pass through multiple downsampling modules to obtain multi-level semantic feature maps. Each downsampling module includes two repeated convolution layers (ReLU activation) and a maximum pooling operation. After passing through each downsampling module, the spatial dimension of the semantic feature map is reduced, thereby extracting higher-level features of the first endoscope image.

[0063] The decoder symmetrically arranged with the encoder includes multiple upsampling modules. The number of the downsampling modules is the same as that of the upsampling modules. In the decoder, each upsampling module is used to restore the spatial resolution of the cross-modal feature map through a transposed convolution layer or an interpolation method, splice the cross-modal feature map with the same spatial dimension as the semantic feature map in the encoder after the spatial resolution is restored, and perform convolution processing on the spliced semantic feature map through a convolution layer. After the cross-modal feature map sequentially passes through multiple upsampling modules, a cross-modal feature map with a spatial resolution consistent with that of the first endoscope image is obtained.

[0064] The cross-modal feature map with the restored spatial resolution output by the decoder is connected to the input image in a residual form to obtain a reconstruction output, which can enable the image processing model to focus on learning the difference between the input and the expected output.

[0065] In step 206, the parameters of the image processing model are adjusted to minimize the difference between the image output by the image processing model and the second endoscope image.

[0066] Optionally, during the model training process, a loss function can be constructed only according to the image output by the image processing model and the second endoscopic image. Alternatively, since the model input includes text and images, forward propagation can also be performed separately for data of the two modalities: the loss of the text modality is calculated according to the image degradation type predicted by the output of the text instruction and the real image degradation type corresponding to the text instruction, and the loss of the image modality is calculated according to the image output by the image processing model and the second endoscopic image; the losses of the two modalities are combined for back propagation, and the model parameters are jointly optimized.

[0067] In the above image processing model training method, the image processing model is trained in combination with the endoscopic image and the related text instruction, so that the image processing model utilizes the prior knowledge provided by the language description to enhance the recognition and processing capability of the model for different degradation types, thereby improving the generalization performance and adaptability of the model, enabling the model to more accurately understand different degradation problems in the image and generate high-quality endoscopic images, thereby solving the problem of being unable to handle multiple endoscopic picture degradation types in the related art.

[0068] In one embodiment, the parameters of the model are adjusted to minimize the difference between the image output by the image processing model and the second endoscopic image, including: inputting the text embedding vector into a classifier, generating a first loss value based on the difference between the image degradation type predicted by the classifier and the image degradation type corresponding to the text embedding vector; generating a second loss value based on the difference between the image output by the image processing model and the second endoscopic image; and adjusting the parameters of the model based on the first loss value and the second loss value.

[0069] The classifier includes but is not limited to a Bayesian classifier, a support vector machine, a convolutional neural network, etc. Optionally, the text embedding vector is input into a three-layer fully connected neural network category classifier to distinguish the image degradation type corresponding to the text embedding vector through the neural network category classifier.

[0070] When calculating the second loss value, the image output by the image processing model and the second endoscopic image correspond to the same first endoscopic image. Optionally, the difference between the image output by the image processing model and the second endoscopic image can be obtained by using a mean square error (MSE), a mean absolute error (MAE), or a perceptual loss (Perceptual Loss) measurement method, and the second loss value can be calculated.

[0071] Optionally, the first loss value and the second loss value are fused into a total loss by weighted summation, and the parameters of the model are adjusted to minimize the total loss, so that the similarity between the image output by the trained image processing model based on the first endoscope image and the text instruction and the second endoscope image is continuously improved, that is, the goal of minimizing the difference between the image output by the image processing model and the second endoscope image is met.

[0072] Optionally, generating the second loss value based on the difference between the image output by the image processing model and the second endoscope image comprises: comparing each pixel in the image output by the image processing model with the pixel in the second endoscope image at the corresponding position to obtain a difference pixel with different pixel values, the difference pixel comprising a first pixel in the image output by the image processing model and a second pixel in the second endoscope image; and generating the second loss value according to the difference between the pixel value of the first pixel and the pixel value of the corresponding second pixel.

[0073] Illustratively, the pixel value of the second pixel in the second endoscope image is taken as a pixel value label, the average of the square differences between the pixel value label and the pixel value of the first pixel is calculated to obtain a mean square error, and a second loss function is constructed according to the mean square error, and the second loss value of the second loss function quantifies the degree of deviation between the model output image and the target second endoscope image. By pixel-level comparison, the image reconstruction loss function is constructed by calculating the pixel-level difference between the predicted image and the real image.

[0074] In this embodiment, the first loss value is established based on the image degradation category, the embedding representation is optimized through the first loss value, and the model network learns the essential differences of various degradation features; the second loss value is established based on the model output image and the real second endoscope image, and the fidelity of image reconstruction is directly optimized through the second loss value; so that the trained image processing model can accurately recognize the text instruction and restore the image degradation phenomenon based on the text instruction, and achieve the goal of minimizing the difference between the image output by the image processing model and the second endoscope image.

[0075] In one embodiment, the channel multiplication is performed on the semantic feature map and the text embedding vector to obtain a cross-modal feature map, comprising: multiplying the semantic feature map and the text embedding vector to obtain a first feature map; assigning weights to the first feature map, so that the weights of elements associated with image degradation phenomena in the first feature map are higher than the weights of elements irrelevant to image degradation phenomena; superimposing the first feature map after assigning weights and the semantic feature map; and performing convolution processing on the superimposed feature map to obtain a cross-modal feature map.

[0076] The first feature map can be dynamically assigned weights based on a self-attention mechanism, so that the image processing model can enhance the recognition ability of related features. Exemplarily, the text embedding vector is multiplied with the semantic feature map of each channel to obtain the first feature map; the first feature map after the weight assignment and the semantic feature map are spliced in the channel dimension to realize the superposition of the first feature map and the semantic feature map; and the superposed feature map is calculated by sliding through one or more convolution kernels to output a cross-modal feature map.

[0077] In this embodiment, the semantic feature map and the text embedding vector are multiplied to realize cross-modal feature modulation; the first feature map after the weight assignment and the semantic feature map are superposed to realize the residual connection of the first feature map and enhance the gradient propagation; and the features of the superposed feature map are further refined by convolution to accurately extract the image degradation features contained in the first endoscope image and the text instruction.

[0078] In one embodiment, the text instruction is converted into a text embedding vector, including: obtaining a text instruction corresponding to an image degradation phenomenon of the first endoscope image; converting the text instruction into a text embedding vector of a preset dimension based on a text encoder; and mapping the text embedding vector based on a linear transformation layer, so that the dimension of the text embedding vector obtained by the mapping corresponds to the semantic feature map.

[0079] The text encoder can be a CLIP text encoder, a BERT encoder, a CNN encoder, etc. The dimension of the text embedding vector obtained by the linear transformation layer mapping can be set based on the configuration of the encoder.

[0080] Optionally, in the case where the image processing model includes multiple encoders, each encoder outputs a multi-level semantic feature map; correspondingly, different dimensions of the text embedding vector need to be multiplied with the multi-level semantic feature map in the channel. In order to ensure the stability of the text embedding vector, an initialized text embedding vector is generated by a pre-trained and parameter-frozen text encoder, and the text embedding vector is connected as a shared representation with the output of the different levels of encoders.

[0081] In this embodiment, the text instruction is converted into a text embedding vector to capture semantic information in the text instruction, which can increase the transfer learning ability of the image processing model.

[0082] In an embodiment, the semantic feature map of the first endoscopic image is extracted based on the encoder, including: performing convolution processing on the first endoscopic image to obtain a second feature map; assigning weights to the second feature map, so that the weights of elements in the second feature map associated with image degradation phenomena are higher than the weights of elements irrelevant to image degradation phenomena; downsampling the second feature map after the weights are assigned; assigning weights to the second feature map after the downsampling, so that the weights of elements in the second feature map after the downsampling associated with image degradation phenomena are higher than the weights of elements irrelevant to image degradation phenomena, and obtaining the semantic feature map.

[0083] In an embodiment, the weights of the second feature map can be assigned based on a self-attention mechanism.

[0084] Optionally, when the encoder includes multiple levels, the steps of downsampling the image after the weights are assigned and assigning weights to the image after the downsampling, so that the weights of elements in the image after the downsampling associated with image degradation phenomena are higher than the weights of elements irrelevant to image degradation phenomena, are repeated at each level. The downsampling includes a convolution layer and a max-pooling operation. Optionally, a nonlinear activation function can also be applied after the convolution layer to introduce nonlinear capabilities and enhance the expressive power of the model.

[0085] In the embodiment, the features are extracted from the first endoscopic image through convolution processing; the relationship between the extracted features is captured by assigning weights to the second feature map, which enhances the model's attention to image degradation phenomena; the size of the feature map is reduced through downsampling, while the abstract level of the features is enhanced, achieving the effect of improving the robustness of the image processing model obtained through training.

[0086] In an embodiment, the training data is obtained, including: simulating image degradation phenomena in endoscopic images without image degradation phenomena to obtain simulated endoscopic images; integrating real endoscopic images with image degradation phenomena and the simulated endoscopic images to obtain the first endoscopic images in the training data.

[0087] Optionally, a batch of clean endoscopic images is first collected. The image degradation phenomena are simulated based on the clean endoscopic images, including: simulating lens stain conditions by adding stain maps of different shapes, sizes and colors on the clean endoscopic images; simulating insufficient or excessive light images by adjusting image brightness, contrast and other parameters using image processing algorithms; generating images with different degrees of motion blur by processing images using specific blur algorithms; and simulating image noise by adding different intensity Gaussian noise. Further, the simulated endoscopic images of different degradation types are taken as the first endoscopic images, the corresponding clean images are taken as the second endoscopic images, and the first endoscopic images and the corresponding second endoscopic images are taken as the training data pairs.

[0088] Optionally, the real endoscope image with image degradation phenomenon is obtained, including: obtaining a large number of real endoscope images from a medical institution, which cover various degradation conditions such as lens stains, uneven lighting, motion blur, noise interference, etc. Further, the obtained large number of real endoscope images are taken as first endoscopic images; the real endoscope images are restored by an existing image restoration method to obtain clean endoscope images which are taken as second endoscopic images; and the first endoscopic images and the corresponding second endoscopic images are taken as training data.

[0089] In this embodiment, the real endoscope images and the simulated endoscope images corresponding to various image degradation phenomena are integrated, which can enrich the training data set and improve the generalization ability and robustness of the image processing model.

[0090] In one embodiment, obtaining the training data further includes: generating text instructions associated with various image degradation types through a large language model.

[0091] Optionally, text descriptions for image degradation problems of smoke, lens stains, reflections, noise, and blur are generated through deepseek or other large language models, and manually screened, and the number of descriptions for each type is at least 5000, and the screened text descriptions are taken as text instructions. It can be understood that the number of text descriptions for each type can be modified according to requirements.

[0092] Exemplarily, the description for smoke includes: "Can you help take out the smoke in this image?";"Please remove the smoke from this image.";"I'd like to have the smoke edited out of this picture.".

[0093] The description for lens stains includes: "Remove the bloodstains from the lens of this image. ";"The lens of this image is dirty. Could you clean it up? ";"The lens of this photo is stained. Please remove the stains. "。

[0094] For the description of the reflection, it includes: "This image has glare; could you help enhance it?" "This image exhibits unwanted reflections. Can you refine it?" "Could you improve this photo to minimize the reflections?"

[0095] For the description of the image noise, it includes: "I need you to remove the noise from this picture." "Enhance this image by reducing the noise present." "Could you please denoise this image?"

[0096] For the description of the image blur, it includes: "This picture needs to be clearer." "This photo is blurry. Please improve its sharpness." "Could you please enhance this image to eliminate the blur?"

[0097] In this embodiment, the texture instruction is generated by a large language model, which is beneficial to improve data diversity, enrich semantic expression, and thus improve model robustness.

[0098] In one embodiment, Figure 3 An image processing model is provided, as shown in Figure 3 The image processing model adopts an improved UNet architecture network model, and the core structure of the network model includes a symmetrically designed encoder (Encoder) and a decoder (Decoder). In addition, the network model also includes a text encoder (CLIP) and a command condition module (Command Condition Block).

[0099] The image processing model comprises a first endoscope image input. An encoder of a symmetrical design is used to process the input of the first endoscope image to obtain a semantic feature map, comprising: receiving a first endoscope image of a known degeneration type, pre-processing through a 3x3 convolution layer to obtain a second feature map, after the second feature map passes through a transformer block, repeating three times of a down sampling module (Down) and an attention module. Each down sampling module comprises two repeated 3x3 convolution layers (ReLU activation) and a 2x2 maximum pooling operation. Each attention module is used to dynamically allocate weights, so that the model can focus on the key areas of the input data (such as texture, edge or specific degeneration features), thereby enhancing the recognition ability of related features. Figure 4 is a schematic diagram of an attention module in this embodiment, as Figure 4 shown, the input data of the attention module is divided into two branches. One branch performs self-attention mechanism calculation, including: calculating three different vectors through convolution, measuring the similarity between them, then performing convolution on the similarity calculation result, and connecting the original input data with the similarity residual error obtained by calculation. The other branch sequentially performs convolution (conv), ReLU activation function, convolution on the input data, to obtain a convolution result; further, the convolution result is processed by average pooling (Avgpool), convolution, ReLU activation function, convolution, Sigmoid activation function, and the convolution result is connected with the data output by the Sigmoid activation function. Finally, the attention module adds and convolves the outputs of the two branches to obtain the output data.

[0100] The image processing model further comprises a text instruction input (Text Prompt). The text instruction is input into the image processing model, and a text embedding vector of a fixed dimension is generated by a pre-trained CLIP text encoder; the parameters of the text embedding vector are frozen during training. The text embedding vector is adapted through different linear transformation layers, so that the dimension of the text embedding vector is consistent with the dimension of the image feature output by each down sampling layer.

[0101] The instruction condition module is used to multiply the semantic feature map output by each down sampling layer and the adapted text embedding vector in the channel, so as to realize cross-modal feature modulation; and through residual connection, the gradient propagation is enhanced to form a fused cross-modal feature map. Figure 5 is a schematic diagram of an instruction condition module in this embodiment, as Figure 5As shown, the text embedding vector is mapped through a Linear (linear transformation layer), and then multiplied by the semantic feature map output by the down-sampling layer to obtain a first feature map; after the first feature map is assigned weights through an attention module, the first feature map is connected in residual with the semantic feature map; and a cross-modal feature map is output through convolution.

[0102] The decoder is used to perform three up-sampling operations on the cross-modal feature map output by the instruction condition module. Among them, three up-sampling modules are included in the decoder. Each up-sampling module restores the resolution of the input data through a transposed convolution layer or an interpolation method, and splices (Skip Connection) it with the feature map output by the same scale encoder, and a 3x3 convolution layer (ReLU activation); finally, the channel number is mapped to the target dimension through a 1x1 convolution layer, and is added to the first endoscope image in residual form to obtain the reconstruction output.

[0103] In this embodiment, the image processing model fuses the endoscope image and the language encoding feature of the text instruction, uses the semantic to guide the description of the degradation type, can more accurately perceive the semantic information of the degradation type, avoids over-optimization or detail loss in the recovery process, and thus outputs more reliable and higher fidelity image results.

[0104] Further, a loss function is constructed, and the image processing model shown in Figure 3 is trained based on the loss function. Figure 3 In the embodiment, the learnable text embedding vector is connected to three different layer positions of the network model as a shared representation. During model training, the text embedding vector can be connected to a three-layer fully connected neural network category classifier, i.e., the text classifier in the above embodiment. The text classifier distinguishes the image degradation category corresponding to the text embedding vector. Optionally, the image degradation type can include one or more of the following: smoke, lens stain, reflection, image noise, or image blur. Based on the output of the text classifier and the true image degradation type corresponding to the text embedding vector, a first loss function is constructed for the image degradation type. This process optimizes the embedding representation, so that the network explicitly learns the essential differences of various degradation features.

[0105] For the image output by the image processing model, a mean square error loss is used to calculate the pixel-level difference between the model output image and the real image (the second endoscope image), and a second loss function is constructed, which directly optimizes the fidelity of image reconstruction.

[0106] According to the first loss function and the second loss function , a total loss function can be constructed:

[0107] ;

[0108] ;

[0109] ;

[0110] in, The total number of training samples includes a first endoscope image, a second endoscope image, and text instructions corresponding to the image degradation phenomenon of the first endoscope image.

[0111] The second endoscopic image pixel value label The first image generated for the model pixel value, The number of categories is 5, where the image degradation types in the training samples include smoke, lens smudges, reflections, image noise, or image blur.

[0112] Text instructions in the training samples In category The real labels on it For classifiers, text instructions In category The predicted probability. for coefficients and for The coefficient. The value and The value can be set according to requirements.

[0113] In this embodiment, a single image processing model trained can efficiently handle multiple types of image degradation without the need to train a separate model for each degradation type, thereby significantly reducing training overhead and memory required for model deployment.

[0114] In one embodiment, such as Figure 6 As shown, an image processing method is provided, applied to an image processing model. The image processing model can be trained based on any of the above-described image processing model training method embodiments, which will not be elaborated upon here. The image processing method includes:

[0115] Step 602: Obtain the target endoscope image exhibiting image degradation and the corresponding text instructions. The target endoscope image is the endoscope image for which image degradation needs to be corrected, and the corresponding text instructions are used to instruct the image processing model to restore the target endoscope image.

[0116] Step 604, input the target endoscope image and the text instruction into the image processing model to obtain a first restored image.

[0117] In this embodiment, the recovery process is dynamically guided by the text instruction, the semantic understanding and model generalization ability are improved through multi-modal fusion, the target endoscope can be personalized restored without professional knowledge, and a high-quality restored image is obtained.

[0118] Further, after obtaining the first restored image, the image processing method further includes: judging whether the first restored image contains image degradation phenomenon; if yes, updating the text instruction based on the first restored image, and inputting the updated text instruction and the first restored image into the model to obtain a second restored image. Optionally, updating the text instruction based on the first restored image includes: generating a new text instruction according to the image degradation phenomenon contained in the first restored image. It can be understood that if the first restored image does not contain image degradation phenomenon, the first restored image does not need to be input into the image processing model again.

[0119] In this embodiment, the text instruction is allowed to be changed to focus on a specific degradation type, increasing the degree of freedom of image restoration and being suitable for diversified real scenarios.

[0120] In one embodiment, the image processing model infers as follows: inputting a target endoscope image with smoke, lens stains, reflections, noise or blur, while providing a text instruction describing the degradation type of the target endoscope image. If there is a single degradation type in the target endoscope image, after inputting the initial degraded target endoscope image and the corresponding text instruction description, the image processing model will output a first restored image with optimized quality. If there are multiple degradation types in the image, the image quality needs to be optimized through an iterative input mechanism. Multiple iterations of the inference process include: each time inputting, replacing or adding a text instruction for a specific degradation type according to the latest output of the model, the model gradually optimizes the specified degradation problem based on the current instruction and the intermediate restored image. For example, after inputting an image containing both smoke and noise, the first iteration is as follows: input the text instruction describing “smoke”, and the model outputs the first restored image with the smoke removed. The second iteration is as follows: using the first restored image output in the last iteration as input, updating the text instruction to the instruction text corresponding to “noise”, and the model further removes the noise to output a second restored image.

[0121] In this embodiment, multiple degradation types are processed by a unified image processing model, and multiple degradation superposition scenarios are optimized through an iterative inference mechanism, and multiple degradation combined images are gradually optimized to ensure high-precision restoration of images in complex superposition scenarios.

[0122] Based on the same inventive concept, the embodiments of the present application also provide an image processing model training device for implementing the image processing model training method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image processing model training device embodiments provided below can be referred to the limitations of the image processing model training method described above, which will not be repeated here.

[0123] In one embodiment, as shown in Figure 7 An image processing model training device is provided, comprising a training data acquisition module, an input module and a training module, wherein:

[0124] The training data acquisition module is configured to acquire training data, wherein the training data comprises a first endoscope image with image degradation phenomenon, a text instruction corresponding to the image degradation phenomenon of the first endoscope image, and a second endoscope image without image degradation phenomenon.

[0125] The input module is configured to input the training data into an image processing model, wherein the image processing model is configured to convert the text instruction into a text embedding vector, extract a semantic feature map of the first endoscope image based on an encoder, perform channel multiplication on the semantic feature map and the text embedding vector to obtain a cross-modal feature map, restore the spatial resolution of the cross-modal feature map to the spatial resolution of the first endoscope image based on a decoder symmetrically arranged with the encoder, and superimpose the cross-modal feature map with restored spatial resolution and the semantic feature map to output a superimposed image.

[0126] The training module is configured to adjust the parameters of the image processing model to minimize the difference between the image output by the image processing model and the second endoscope image.

[0127] In one embodiment, the training module adjusts the parameters of the model to minimize the difference between the image output by the image processing model and the second endoscope image, comprising: inputting the text embedding vector into a classifier, generating a first loss value based on the difference between the image degradation type predicted by the classifier and the image degradation type corresponding to the text embedding vector; generating a second loss value based on the difference between the image output by the image processing model and the corresponding second endoscope image; and adjusting the parameters of the model based on the first loss value and the second loss value.

[0128] Optionally, the training module generates the second loss value based on a difference between the image output by the image processing model and the second endoscopic image includes: comparing each pixel in the image output by the image processing model with a pixel at a corresponding position in the second endoscopic image to obtain a difference pixel with different pixel values, the difference pixel including a first pixel in the image output by the image processing model and a second pixel in the second endoscopic image; and generating the second loss value according to a mean square error between a pixel value of the first pixel and a pixel value of the corresponding second pixel.

[0129] In one of the embodiments, the input module performs channel multiplication on the semantic feature map and the text embedding vector to obtain the cross-modal feature map, including: multiplying the semantic feature map and the text embedding vector to obtain a first feature map; assigning weights to the first feature map, so that the weights of elements in the first feature map associated with the image degradation phenomenon are higher than the weights of elements irrelevant to the image degradation phenomenon; superimposing the first feature map after the weights are assigned and the semantic feature map; and performing convolution processing on the superimposed feature map to obtain the cross-modal feature map.

[0130] In one of the embodiments, the input module converts the text instruction into a text embedding vector, including: obtaining a text instruction corresponding to the image degradation phenomenon of the first endoscopic image; converting the text instruction into a text embedding vector of a preset dimension based on a text encoder; and mapping the text embedding vector based on a linear transformation layer, so that the dimension of the text embedding vector obtained by the mapping corresponds to the semantic feature map.

[0131] In one of the embodiments, the input module extracts a semantic feature map of the first endoscopic image based on an encoder, including: performing convolution processing on the first endoscopic image to obtain a second feature map; assigning weights to the second feature map, so that the weights of elements in the second feature map associated with the image degradation phenomenon are higher than the weights of elements irrelevant to the image degradation phenomenon; downsampling the second feature map after the weights are assigned; assigning weights to the second feature map after the downsampling, so that the weights of elements in the second feature map after the downsampling associated with the image degradation phenomenon are higher than the weights of elements irrelevant to the image degradation phenomenon, and obtaining the semantic feature map.

[0132] In one of the embodiments, the input module obtains training data, including: simulating an image degradation phenomenon in an endoscopic image without the image degradation phenomenon to obtain a simulated endoscopic image; and integrating a real endoscopic image with the image degradation phenomenon and the simulated endoscopic image to obtain the first endoscopic image in the training data.

[0133] Each module in the aforementioned image processing model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0134] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing model training method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0135] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0138] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the above method embodiments.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned method embodiments. Any reference to a memory, database or other medium used in the embodiments provided in the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0140] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0141] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image processing model training method, characterized in that, The method comprises: obtaining training data, the training data comprising a first endoscope image with image degradation phenomenon, a text instruction corresponding to the image degradation phenomenon of the first endoscope image, and a second endoscope image without image degradation phenomenon; inputting the training data into an image processing model; wherein the image processing model is used to convert the text instruction into a text embedding vector; an encoder is used to extract a semantic feature map of the first endoscope image; the semantic feature map and the text embedding vector are multiplied in the channel to obtain a cross-modal feature map; a decoder symmetrically arranged with the encoder is used to restore the spatial resolution of the cross-modal feature map to the spatial resolution of the first endoscope image; the cross-modal feature map with restored spatial resolution and the semantic feature map are superimposed to output the superimposed image; adjusting the parameters of the image processing model to minimize the difference between the image output by the image processing model and the second endoscope image.

2. The image processing model training method of claim 1, wherein, Adjusting the parameters of the model to minimize the difference between the image output by the image processing model and the second endoscope image comprises: inputting the text embedding vector into a classifier to generate a first loss value based on the difference between the image degradation type predicted by the classifier and the image degradation type corresponding to the text embedding vector; generating a second loss value based on the difference between the image output by the image processing model and the corresponding second endoscope image; adjusting the parameters of the model based on the first loss value and the second loss value.

3. The image processing model training method of claim 2, wherein, Generating a second loss value based on the difference between the image output by the image processing model and the second endoscope image comprises: comparing each pixel in the image output by the image processing model with the pixel in the second endoscope image at the corresponding position to obtain a difference pixel with different pixel values, the difference pixel comprising a first pixel in the image output by the image processing model and a second pixel in the second endoscope image; generating the second loss value based on the mean square error between the pixel value of the first pixel and the pixel value of the corresponding second pixel.

4. The image processing model training method of claim 1, wherein, Multiplying the semantic feature map and the text embedding vector in the channel to obtain a cross-modal feature map comprises: multiplying the semantic feature map and the text embedding vector to obtain a first feature map; assigning weights to the first feature map so that the weights of elements associated with image degradation phenomenon in the first feature map are higher than the weights of elements unrelated to image degradation phenomenon; superimposing the first feature map with assigned weights and the semantic feature map; performing convolution processing on the superimposed feature map to obtain the cross-modal feature map.

5. The image processing model training method of claim 1, wherein, Converting the text instruction into a text embedding vector comprises: obtaining a text instruction corresponding to the image degradation phenomenon of the first endoscope image; converting the text instruction into a text embedding vector with a preset dimension based on a text encoder; mapping the text embedding vector based on a linear transformation layer so that the dimension of the mapped text embedding vector corresponds to the semantic feature map.

6. The image processing model training method of claim 1, wherein, Extracting a semantic feature map of the first endoscope image based on an encoder comprises: performing convolution processing on the first endoscopic image to obtain a second feature map; assigning weights to the second feature map, so that the weights of elements in the second feature map associated with image degradation phenomena are higher than the weights of elements not associated with image degradation phenomena; performing down-sampling on the second feature map to which the weights are assigned; assigning weights to the down-sampled second feature map, so that the weights of elements in the down-sampled second feature map associated with image degradation phenomena are higher than the weights of elements not associated with image degradation phenomena, and obtaining the semantic feature map.

7. The image processing model training method of claim 1, wherein, obtaining training data, including: simulating image degradation phenomena in endoscopic images without image degradation phenomena to obtain simulated endoscopic images; integrating real endoscopic images with image degradation phenomena and the simulated endoscopic images to obtain first endoscopic images in the training data.

8. An image processing method characterized by, applying to an image processing model trained based on the image processing model training method of any one of claims 1 to 7; the image processing method comprising: obtaining a target endoscopic image with image degradation phenomena and a text instruction corresponding to the target endoscopic image; inputting the target endoscopic image and the text instruction into the image processing model to obtain a first restored image.

9. The image processing method of claim 8, wherein, After obtaining the first restored image, the image processing method further comprises: determining whether the first restored image contains image degradation phenomena; if yes, updating the text instruction based on the first restored image, and inputting the updated text instruction and the first restored image into the model to obtain a second restored image.

10. An image processing model training apparatus, characterized by comprising: The device comprises: a training data acquisition module configured to obtain training data, the training data comprising first endoscopic images with image degradation phenomena, text instructions corresponding to image degradation phenomena of the first endoscopic images, and second endoscopic images without image degradation phenomena; an input module configured to input the training data into an image processing model; wherein the image processing model is configured to convert the text instructions into text embedding vectors, extract semantic feature maps of the first endoscopic images based on an encoder, perform channel multiplication on the semantic feature maps and the text embedding vectors to obtain cross-modal feature maps, restore the spatial resolution of the cross-modal feature maps to the spatial resolution of the first endoscopic images based on a decoder symmetrically arranged with the encoder, and superimpose the cross-modal feature maps with restored spatial resolution and the semantic feature maps to output superimposed images; a training module configured to adjust parameters of the image processing model to minimize the difference between images output by the image processing model and the second endoscopic images.

Images