Model training method and device based on image enhancement

Abstract

The invention provides a model training method and device based on image enhancement. The method comprises the following steps: acquiring a scanning image and a noise image; executing preprocessing operation on the scanning image according to the noise image, and determining a training image set; inputting the training images in the training image set into a generator in a preset generative adversarial network model to obtain enhanced images corresponding to the training images; inputting the enhanced image and a synthesized high-definition image in the training image set into a discriminator in a generative adversarial network model to obtain a discrimination result and a loss function value; and updating parameters of the discriminator and parameters of the generator according to the loss function value so as to train and obtain a target image enhancement model. Therefore, by superposing the noise image to the scanning image and then inputting the image to the generator for training and optimization, the stability and robustness of the model are improved. And the enhanced image and the synthesized high-definition image are discriminated through the discriminator to repeatedly train the model, so that the image enhancement performance of the generator and the discrimination capability of the discriminator are improved.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a model training method and device based on image enhancement. Background Art

[0002] At present, image scanning technology has been widely used in clinical diagnosis in the medical field.

[0003] However, in actual application scenarios, the scanned images obtained by image scanning technology will be affected by image noise, thereby reducing the image quality of the scanned images. Although traditional image denoising algorithms can effectively suppress image noise, they have the problems of high model complexity and poor algorithm stability, and the denoised scanned images lack image details, resulting in poor practicality of the processed scanned images. Summary of the invention

[0004] The embodiment of the present application provides a model training method and device based on image enhancement, which improves the stability and robustness of the model by superimposing a noise image on a scanned image and then inputting it into a generator for training and optimization. The enhanced image and the synthesized high-definition image are discriminated by a discriminator to repeatedly train the model in order to improve the image enhancement performance of the generator and the discrimination ability of the discriminator.

[0005] In a first aspect, an embodiment of the present application provides a model training method based on image enhancement, the method comprising:

[0006] Acquire a scanned image and a noise image, wherein the scanned image is a medical image acquired by scanning a target position of a detection object, and the noise image is a medical image acquired by scanning a background where no detection object exists;

[0007] According to the noise image, a preprocessing operation is performed on the scanned image to determine a training image set;

[0008] Inputting a training image in the training image set into a generator in a preset generative adversarial network model to obtain an enhanced image corresponding to the training image; and,

[0009] Input the enhanced image and the synthetic high-definition image in the training image set into the discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result, and the discrimination result is used to indicate the probability that the discriminator determines that the enhanced image belongs to a real high-definition image;

[0010] The parameters of the discriminator and the generator are updated according to the loss function value to train the generative adversarial network model to obtain the target image enhancement model, which is the generative adversarial network model after training.

[0011] In a second aspect, an embodiment of the present application provides a model training device based on image enhancement, the device comprising:

[0012] An image acquisition unit, used to acquire a scanned image and a noise image, wherein the scanned image is a medical image acquired by scanning a target position of a detection object, and the noise image is a medical image acquired by scanning a background where no detection object exists;

[0013] A preprocessing unit, used for performing a preprocessing operation on the scanned image according to the noise image to determine a training image set;

[0014] A first training unit is used to input a training image in a training image set into a generator in a preset generative adversarial network model to obtain an enhanced image corresponding to the training image; and input the enhanced image and the synthetic high-definition image in the training image set into a discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result, wherein the discrimination result is used to indicate the probability that the discriminator determines that the enhanced image belongs to a real high-definition image;

[0015] The second training unit is used to update the parameters of the discriminator and the parameters of the generator according to the loss function value to train the generative adversarial network model to obtain a target image enhancement model, which is the generative adversarial network model after training.

[0016] In a third aspect, an embodiment of the present application provides a server, comprising a processor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps in the first aspect of the embodiment of the present application when the computer program / instruction is executed by a processor.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement some or all of the steps described in the first aspect of the embodiment of the present application.

[0019] It can be seen that in the embodiment of the present application, the server obtains the noise image and the scanned image taken by the medical scanning device, and then superimposes the noise image on the scanned image, and then inputs it into the generator for training and optimization, so as to improve the stability and robustness of the model. In addition, the enhanced image and the synthesized high-definition image are distinguished by the discriminator to repeatedly train the model, thereby improving the image enhancement performance of the generator and the discrimination ability of the discriminator. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a structural block diagram of an image enhancement system provided in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a model training method based on image enhancement provided in an embodiment of the present application;

[0023] Figure 3 It is a structural diagram of a generative adversarial network model training provided in an embodiment of the present application;

[0024] Figure 4 It is a structural diagram of another generative adversarial network model training provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of the structure of an attention module provided in an embodiment of the present application;

[0026] Figure 6a An image schematic diagram of an OCT scan image provided by an embodiment of the present application;

[0027] Figure 6b is an image schematic diagram of a synthesized high-definition image provided by an embodiment of the present application;

[0028] Figure 6c is an image schematic diagram of an enhanced image provided by an embodiment of the present application;

[0029] Figure 6d is an image schematic diagram of an OCT lesion image provided in an embodiment of the present application;

[0030] Figure 6e This is an image schematic diagram of a synthetic high-definition image of an OCT lesion image provided in an embodiment of the present application;

[0031] Figure 6f is a schematic diagram of an enhanced image of an OCT lesion image provided in an embodiment of the present application;

[0032] Figure 7 It is a block diagram of the functional units of a model training device based on image enhancement provided in an embodiment of the present application;

[0033] Figure 8It is a block diagram of the functional units of another model training device based on image enhancement provided in an embodiment of the present application;

[0034] Fig. 9 It is a structural block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] See also Figure 1 , Figure 1 is a structural block diagram of an image enhancement system provided in an embodiment of the present application. Figure 1As shown, the image enhancement system 100 includes a server 110, a medical scanning device 120 and a display device 130. The medical scanning device 120 is used to scan the detection object 140 to obtain a scanned image, and scan the background to obtain a noise image. The medical scanning device 120 refers to one or more medical scanning devices of different instrument types that apply the same image scanning technology. The medical scanning device 120 is connected to the server 110 for communication, and the display device 130 is connected to the server 110 for communication. The server 110 obtains the scanned image and the noise image scanned by the medical scanning device 120, and then performs preprocessing operations on the scanned image and the noise image to obtain a training image set. The server 110 uses the training image set to train the generator and the discriminator in the preset generative adversarial network model until the target image enhancement model, that is, the generative adversarial network model after training is obtained. Afterwards, when the medical scanning device 120 scans the test object 140 to obtain a new scanned image, the server 110 processes the new scanned image obtained by the medical scanning device 120, and enhances the scanned image through the target image enhancement model to generate an enhanced image. The server 110 sends the image data of the enhanced image to the display device 130 to display the enhanced image. Doctors and patients can view the enhanced image through the display device 130 to make clinical diagnosis. One image enhancement system 100 corresponds to one server 110, one server 110 corresponds to one or more medical scanning devices 120, and each medical scanning device 120 corresponds to one display device 130.

[0039] Based on this, an embodiment of the present application provides a model training method based on image enhancement, and the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0040] See also Figure 2 , Figure 2 1 is a flow chart of a model training method based on image enhancement provided by an embodiment of the present application. The method is applied to a server 110 in an image enhancement system 100. The image enhancement system 100 also includes a medical scanning device 120 and a display device 130. The server 110 is respectively connected to the medical scanning device 120 and the display device 130 in communication. The method includes:

[0041] Step S201, obtaining a scanned image and a noise image.

[0042] The scanned image is a medical image acquired by scanning a target position of the detection object, and the noise image is a medical image acquired by scanning a background where no detection object exists.

[0043] Among them, the image scanning technology involved in the medical image acquired by the server in this application can be any image scanning technology commonly used in clinical diagnosis, such as X-ray imaging, computed tomography (CT), magnetic resonance imaging, ultrasound imaging, optical coherence tomography (OCT), etc. Among them, the OCT image is an image obtained by optical coherence tomography technology, which has the characteristics of high resolution, fast imaging and non-invasive (ophthalmic OCT) or invasive (cardiovascular OCT). It plays an important role in the diagnosis and treatment of ophthalmic and cardiovascular diseases. In the subsequent description section of this application, the model training method based on image enhancement provided by this application will be described by taking OCT technology as an example. Taking OCT as an example, OCT has been widely used in clinical diagnosis due to its non-invasive, real-time and high-resolution characteristics. However, the inherent speckle noise (the noise in the image acquired by other image scanning technologies is not necessarily speckle noise) seriously reduces the image quality and may destroy the fine structure in OCT, thereby affecting the diagnosis result. Speckle noise is actually a wave interference phenomenon. When coherent imaging systems such as laser, radar and ultrasound are irradiated onto the surface of an object, these coherent imaging systems will obtain images from these structures because the surface or internal interface of the object is usually extremely rough on the scale of wavelength. In these images, the signal waves between a large number of scatterers will interfere with each other, thereby generating speckle. Speckle forms a signal with irregular alternation of bright spots and dark spots in the image, which is called "speckle noise". It is specifically manifested as high-intensity granular spots in the image, which will reduce the resolution and contrast of the image, seriously affecting the imaging quality and the accuracy of subsequent image analysis.

[0044] The scanned image is a medical image acquired by a relevant technician (a doctor operating a medical scanning device) by scanning the target position of the test object. The scanned image here can be understood as an image directly acquired by the medical scanning device, that is, an original image without additional image processing. Since OCT images acquired by different types of medical scanning devices will be affected by speckle noise to varying degrees, the medical scanning device can acquire an image corresponding to the speckle noise that exists due to the device itself or the principle of image scanning technology by means of air shooting, that is, a noise image acquired by scanning a background without a test object.

[0045] In one possible embodiment, acquiring a scanned image and a noise image includes: scanning a detection object by a first medical scanning device to acquire an initial acquisition image; determining an initial position of an imaging probe of the first medical scanning device in a horizontal direction; according to the initial position, controlling the first medical scanning device to repeatedly scan the detection object to acquire a preset number of subsequent acquisition images; integrating the initial acquisition image and the subsequent acquisition images to determine a scanned image; scanning a background in which no detection object exists in each acquisition mode by a second medical scanning device to acquire an aerial image; and integrating all the aerial images to determine a noise image.

[0046] The horizontal direction is a direction perpendicular to the penetration direction of the light beam corresponding to the imaging probe, and the second medical scanning device includes one or more medical scanning devices of different instrument types from the first medical scanning device.

[0047] Among them, in order to maintain the consistency of the scanned image that needs to be processed, the scanned image is obtained by scanning and collecting the same horizontal position (i.e., the above-mentioned target position) of the detection object through a medical scanning device of the same instrument type (i.e., the first medical scanning device). The method for determining the same horizontal position is that when the relevant technician uses the first medical scanning device to scan the target position of the detection object to obtain the initial acquisition image, the initial position of the imaging probe in the horizontal direction when the image is acquired is determined. Then, the imaging probe is fixed at the initial position to repeatedly scan the target position of the detection object. The principle of repeated scanning is that increasing the number of repeated scans can improve the signal-to-noise ratio and can improve the quality of the OCT synthetic high-definition image obtained after subsequent server processing. It should be noted that in actual application scenarios, the operation of repeated scanning will also increase the scanning time, and the increase in scanning time will increase the risk of eye movement noise (taking the acquisition of ophthalmic OCT images as an example, eye movement noise is the random fluctuation around the fixation area caused by the patient's eye movement during the scanning process), thereby affecting the clarity and accuracy of the OCT high-definition image. This embodiment comprehensively considers the number of repeated scans and the image quality during scanning time. Preferably, the number of repeated scans is set to 20, that is, the number of subsequent acquired images is 19. The time consumed for a single scan is related to the instrument type of the first medical scanning device. Different instrument types have different times for obtaining scanned images under the same image scanning technology.

[0048] The server obtains the aerial images (i.e., the scanned images obtained by scanning the background without the detection object) collected by the second medical scanning device in each acquisition mode (i.e., shooting mode), and then integrates all the aerial images to obtain the noise image. The second medical scanning device is one or more medical scanning devices of different instrument types from the first medical scanning device. The principle of this step is that due to hardware differences, different instruments generate different noises. Different shooting modes of the same instrument generate different noises. The fast acquisition mode generates large noise, and the slow acquisition mode generates small noise. Therefore, in order to improve the robustness of the model, different instruments and different shooting modes are adapted, and noise images of different instruments are collected for subsequent preprocessing of the scanned images and to improve the effect of training the generative adversarial network model.

[0049] It can be seen that in this example, when the server acquires the scanned image, increasing the number of repeated scans can improve the signal-to-noise ratio and the quality of the subsequently acquired synthetic high-definition image. In addition, in order to improve the robustness of the model, it can adapt to different instruments and different acquisition models, thereby acquiring various types of aerial images for subsequent image processing operations and model training processes.

[0050] Step S202: performing a preprocessing operation on the scanned image according to the noise image to determine a training image set.

[0051] The training image set includes synthetic high-definition images and training images (i.e., preprocessed scanned images), wherein the training images can be understood as training samples, and the synthetic high-definition images can be understood as comparison samples. The process of model training is to enable the generator in the preset generative adversarial network model to obtain processed training samples by performing image enhancement operations on the training samples, and then discriminate the processed training samples and comparison samples through the discriminator, and then update the parameters of the generator and the discriminator until the processed training samples output by the generator cannot be distinguished by the discriminator, that is, the image-enhanced training images (scanned images obtained by simulating different types of medical scanning devices under different acquisition modes) are difficult to distinguish from the synthetic high-definition images. Subsequently, the trained generative adversarial network model can be used to perform image enhancement operations on the original OCT images obtained by the medical scanning device in actual application scenarios to eliminate speckle noise in the images, improve image quality, and improve the practicality of the images.

[0052] In a possible embodiment, the training image set includes multiple reference training pairs, each of which includes a single training image and a single synthetic high-definition image; a preprocessing operation is performed on the scanned image according to the noise image to determine the training image set, including: performing a high-definition image generation operation on the initial acquisition image and the subsequent acquisition image in the scanned image to generate a synthetic high-definition image to be processed for the target position; combining the synthetic high-definition image to be processed and each initial acquisition image or the subsequent acquisition image in the scanned image into multiple initial training image pairs respectively; performing a slider cropping operation on each initial training image pair according to a preset sliding step size and a sliding window size to obtain multiple sub-image pairs, and performing a slider cropping operation on the noise image to obtain multiple noise image blocks; determining a single high-definition image block whose pixel average value is greater than a preset pixel threshold as a processed synthetic high-definition image, and determining a sub-image pair containing the processed synthetic high-definition image as a reference training pair; performing noise superposition on a single scanned image block and any one noise image block in each reference training pair respectively, and performing a simulation processing operation on the scanned image block after noise superposition to obtain a training image in each reference training pair.

[0053] Among them, the scanned images in any two initial training image pairs are different from each other, each sub-image pair includes a single scanned image block and a single high-definition image block, and the simulation processing operation is used to perform image changes on the scanned image blocks after noise superposition to simulate the scanned images collected by the first medical scanning device under different scanning environments.

[0054] The preprocessing operation includes performing a high-definition image generation operation based on the initial acquisition image and the subsequent acquisition image in the scanned image to generate a synthetic high-definition image, and forming the synthetic high-definition image and the scanned image into a training data pair, and using a slider window to crop the training pair to perform preprocessing at the same time.

[0055] Exemplarily, the server uses a preset sliding step size (128x128) and a sliding window size (256x256) to perform a slider cropping operation on each initial training image pair to perform slider cropping on a single OCT image and an OCT high-definition image. The training pairs cropped into small blocks (i.e., multiple sub-image pairs) are further processed, and the training pairs whose pixel mean values ​​corresponding to the sub-image pairs (i.e., OCT high-definition image blocks) are selected are greater than a preset pixel threshold (preferably, the threshold is 20). Among them, the window size, sliding step size, and preset pixel threshold of the slider cropping are optional instances. When training with the original image size, the image background accounts for too large a proportion, resulting in the model denoising effect being too strong and ignoring the detail information in the OCT. The slider window and pixel mean threshold can avoid the situation where the image background accounts for too large a proportion in the training data. Moreover, the setting of the window size and the slider step size affects the amount of training data and the training speed. Taking all factors into consideration, in this example, the window size is set to 256x256, the slider step size is 128x128, and the pixel mean threshold is 20.

[0056] Furthermore, after obtaining a 256x256 image block training data pair (including a corresponding single scanned image block and a single high-definition image block), a 256x256 noise image block is randomly cropped from the collected noise image and superimposed with a 256x256 single OCT image block (i.e., a single scanned image block) to simulate OCT images of different instruments or different acquisition modes. In order to help the model learn more robust features, the scanned image blocks after noise superposition are further processed and simulated preprocessing is added. The simulation preprocessing operations include rotation, vertical translation, horizontal translation, scaling, shearing and other operations. Specifically, the rotation angle range is -10° to 10°, simulating different shooting angles and tilt angles in the OCT images of patients with high myopia; the scaling ratio range is 0.8 to 1.2, simulating OCT images of different resolutions; shearing means that the image will be sheared up to 10° along the X-axis or Y-axis to simulate the non-rigid deformation caused by eye movement during shooting. Among them, by simulating the preprocessing operation, the server can perform image changes on the scanned image blocks after noise superposition to simulate the following aspects: 1. Changes in shooting angle and position: Through random rotation, vertical translation and horizontal translation, OCT images acquired at different shooting angles and positions can be simulated. This helps the model learn image features under different viewing angles, thereby improving the ability to recognize images under different shooting conditions. 2. Changes in resolution: Through random scaling, OCT images of different resolutions can be simulated. In practical applications, the resolution of images may vary due to factors such as equipment, shooting conditions, or compression during image transmission. Through random scaling operations, the model can learn feature representations that remain stable at different resolutions. 3. Non-rigid deformation: Through random shearing, non-rigid deformation caused by eye movement or other factors during shooting can be simulated. This deformation may distort or deform the structure in the image, thereby affecting the recognition accuracy of the model. Through random shearing operations, the model can learn the ability to accurately recognize image features under deformation conditions.

[0057] It can be seen that in this example, the server can effectively simulate scanning images under different instruments, different acquisition modes, and different background environments by preprocessing operations to superimpose various types of noise images, thereby improving the comprehensiveness and stability of the network model and improving the generalization ability of the model.

[0058] In a possible embodiment, a high-definition image generation operation is performed on an initial acquisition image and subsequent acquisition images in a scanned image to generate a synthetic high-definition image to be processed for a target position, including: determining any subsequent acquisition image or the initial acquisition image in the scanned image as a reference image; performing a registration operation on the remaining scanned images except the reference image through affine transformation to align the reference image; determining multiple pixel averages according to the pixel value of each pixel position corresponding to the registered scanned image; generating an average image according to the pixel average corresponding to each pixel position; dividing the average image into multiple reference image blocks of preset sizes, and calculating the grayscale histogram of each reference image block; performing a histogram equalization operation on each reference image block according to a preset contrast threshold and grayscale histogram to obtain a processed reference image block; and determining the synthetic high-definition image to be processed according to the processed reference image block.

[0059] Among them, each pixel average value corresponds to a pixel position.

[0060] Among them, an image is randomly selected from multiple collected OCT images (i.e., subsequent collected images or initial collected images) as a reference image, and the other images are registered through affine transformation. The multiple registered OCT images are averaged and subjected to contrast-limited adaptive histogram equalization processing to obtain an OCT high-definition image. The contrast-limited adaptive histogram equalization processing specifically includes affine transformation, pixel averaging operation, image blocking operation and histogram equalization operation. Among them, the affine transformation registration operation ensures the consistency of all scanned images in spatial position, providing a basis for subsequent processing. The pixel averaging operation can reduce image noise and differences and improve the overall quality of the image by calculating the pixel average and generating an average image. The histogram equalization operation can enhance the contrast of the image, make the details in the image clearer, and improve the visual effect of the image. The synthetic high-definition image generated after the scanned image is processed by the server combines the information of multiple scanned images, has higher resolution and better visual effects, and is suitable for various image processing and analysis tasks. In this way, adaptive enhancement of the local contrast of the image is achieved, while avoiding noise amplification, further improving the OCT image quality to obtain a synthetic high-definition image, enhancing the image contrast, making the lesion area clearer, and helping doctors' diagnosis.

[0061] It can be seen that in this example, the server obtains high-quality training images and enhances the contrast of images by performing registration operations on the scanned images and adaptively enhancing the local contrast of the images while avoiding noise amplification.

[0062] Step S203: input the training images in the training image set into the generator in the preset generative adversarial network model to obtain enhanced images corresponding to the training images.

[0063] Among them, the generative adversarial network model is a deep learning model, which mainly consists of two parts: the generator and the discriminator. Among them, the generator is specifically responsible for generating artificial samples that are as close to the real data distribution as possible. The generator in the traditional generative adversarial network model receives a random noise (such as Gaussian noise or uniform noise) as input, and tries to generate samples (i.e., enhanced images) similar to the real data (i.e., synthetic high-definition images) through the noise. As the training progresses, the generator continuously improves its generation ability and can generate data that is closer and closer to the real data. Discriminator: Responsible for judging whether the input data (i.e., enhanced image) is real or generated by the generator. It receives a sample (i.e., the enhanced scanned image) as input and outputs a probability value between 0 and 1, indicating the probability that the sample is a real sample (i.e., synthetic high-definition image). The goal of the discriminator is to distinguish between real data and generated data as accurately as possible.

[0064] Among them, the design of the generator is different from the traditional generative adversarial network model in that the traditional generative adversarial network model is usually a conditional generative adversarial network model, and the conditional generative adversarial network generator accepts two inputs: one is random noise (such as Gaussian noise or uniform noise), and the other is conditional information (such as category labels, text descriptions, images, etc.). Directly using the conditional generative adversarial network generator to generate enhanced images may cause instability. Taking OCT images as an example, the noise in OCT images is not simple Gaussian noise or uniform noise. The noise in OCT images is a tomographic image produced by the coherent interference of the reflected light and the reference light after scattering when light propagates in biological tissues, which is called speckle noise. In addition, there are many types of OCT image lesions, such as macular hole epiretinal membrane, central serous chorioretinopathy, macular edema, diabetic maculopathy, etc. There are many forms of lesions of the same type. Inputting random noise into the generator may cause the generator to generate false information, and in severe cases may cause doctors to misjudge. In order to improve the stability of the generator, in the training method of the generative adversarial network model provided in the present application, a pre-step is added when inputting the training image into the generative adversarial network model, that is, speckle noise is superimposed on the scanned image to obtain a training image set, which is then input into the generator for training, so as to train the generator to learn the real OCT image and maintain the consistency of the generated enhanced image with the scanned image in the real application scenario.

[0065] Step S204: input the enhanced image and the synthetic high-definition image in the training image set into the discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result.

[0066] The discrimination result is used to indicate the probability that the discriminator determines that the enhanced image belongs to a true high-definition image.

[0067] Among them, the iterative optimization algorithm (Adaptive Moment Estimation, Adam) can be used in the process of model training. It is an iterative optimization algorithm used to minimize the loss function during neural network training. It combines dynamic learning rate and smoothing techniques to use momentum by using the moving average of the gradient, thereby helping the model converge to the global minimum faster. The Adam optimizer scales the learning rate by calculating the moving average of the square of the gradient, which helps to deal with non-stationary targets and adaptive learning rates, so that the model can better adapt to changes in data during training. The learning rate is 0.0002, the amount of data for each training is 50, and the number of training times is 200.

[0068] In one possible embodiment, the enhanced image and the synthetic high-definition image in the training image set are input into the discriminator in the generative adversarial network model to obtain the discrimination result and the loss function value corresponding to the discrimination result, including: inputting the enhanced image and the synthetic high-definition image into the discriminative encoder to obtain feature information, a first scalar value and a second scalar value of the enhanced image and the synthetic high-definition image; inputting the feature information into the discriminative decoder to obtain a third scalar value and a fourth scalar value; determining the discrimination result according to the first scalar value, the second scalar value, the third scalar value and the fourth scalar value; determining the encoding loss value according to the first scalar value, the second scalar value and the preset encoder loss function, and determining the decoding loss value according to the third scalar value, the fourth scalar value and the preset decoder loss function; determining the first function value according to the preset discriminator loss function, the encoding loss value and the decoding loss value; and determining the second function value according to the preset generator loss function, the second scalar value and the fourth scalar value.

[0069] Among them, the discriminator includes a discriminative encoder and a discriminative decoder, and the loss function value includes a first function value and a second function value. The first function value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image from the synthesized high-definition image, and the second function value is used to characterize the error size between the enhanced image generated by the generator and the synthesized high-definition image.

[0070] Among them, the first scalar value is used to indicate the probability that the discriminant encoder determines that the synthesized high-definition image belongs to the real high-definition image, the second scalar value is used to indicate the probability that the discriminant encoder determines that the enhanced image belongs to the real high-definition image, the third scalar value is used to indicate the probability that the corresponding position of each pixel of the synthesized high-definition image determined by the discriminant decoder belongs to the high-definition image detail, the fourth scalar value is used to indicate the probability that the corresponding position of each pixel of the enhanced image determined by the discriminant decoder belongs to the high-definition image detail, the encoding loss value is used to characterize the accuracy of the discriminator in distinguishing between the enhanced image and the synthesized high-definition image at the overall image level, the decoding loss value is used to characterize the accuracy of the discriminator in distinguishing between the enhanced image and the synthesized high-definition image at the image detail level, and the discriminator loss function includes a first variable coefficient corresponding to the first function value and a second variable coefficient corresponding to the second function value.

[0071] For example, please refer to Figure 3 , Figure 3 Schematic diagram of a structure of a generative adversarial network model training provided in an embodiment of the present application. Figure 3 The loss function value, first function value, second function value, first scalar value, second scalar value, third scalar value, fourth scalar value, decoding loss value, and encoding loss value mentioned in this embodiment are described below through formula calculation.

[0072] Among them, in order to improve the effect of the generator, the model structure of the generator in this embodiment can be composed of two parts: an encoder and a decoder. The encoder is mainly responsible for extracting the feature information of the input image (i.e., the training image in any reference training pair in the training image set). Through the combination of convolutional layers and pooling layers, the resolution of the image is gradually reduced, and the number of channels of the feature map is increased to obtain high-level semantic features. The decoder is mainly responsible for restoring the extracted feature information to the original image size, and gradually restores the resolution of the image through the combination of upsampling layers and convolutional layers, and splices (jump links) with the feature maps of the corresponding layers of the encoder to fuse feature information at different levels. Through the jump link structure, the model structure of the generator can fuse the high-resolution features in the encoder with the low-resolution features in the decoder, thereby retaining more detail information and improving the accuracy of segmentation.

[0073] Among them, the second function value L of the generator is obtained G The generator loss function is:

[0074] L G = -E[logD enc (G(x))+∑ i,j log[D dec (G(x))] i,j ],

[0075] Among them, G represents the generator, D enc represents the encoder of the discriminator (i.e., discriminative encoder), D dec represents the decoder of the discriminator (i.e., the discriminative decoder), x represents the input single training image, G(x) represents the enhanced image corresponding to x, logD enc (G(x)) represents a second scalar value, used to indicate the probability that the discriminative encoder determines that the enhanced image belongs to the real high-definition image, ∑ i,j log[D dec (G(x))] i,j represents a fourth scalar value, which is used to indicate the probability that the enhanced image determined by the discriminant decoder belongs to the high-definition image detail at the corresponding position of each pixel point. E is represented by the mathematical expectation after the second scalar value and the fourth scalar value are added. The learning goal of G is to pay attention to both structural information and detail information at the same time to generate a more realistic OCT enhanced image.

[0076] Based on the model structure of the generator mentioned above, the discriminator in the generative adversarial network model can also adopt a model structure similar to the generator to discriminate the enhanced image output by the generator. However, the discriminator adds a convolutional layer at the end of the discriminative encoder as a whole structure discriminant of the generated image.

[0077] Among them, the first function value L of the discriminator is obtained D The discriminator loss function is:

[0078]

[0079] in, is the encoder loss value, is the decoder loss value, α and β are the coefficients of the encoder loss function (i.e. the first variable coefficient corresponding to the first function value) and the decoder loss function (i.e. the first variable coefficient corresponding to the second function value), respectively, which control the smoothness and detail richness of the generated image. In this example, α=1, β=1.

[0080] Obtain the encoder loss value The encoder loss function is:

[0081]

[0082] Obtain the decoder loss value The decoder loss function is:

[0083]

[0084] Among them, y represents the synthetic high-definition image, and y constitutes the training data pair (x, y), D enc(y) and D enc (G(x)) represents the decision of the overall image encoder, [D dec (y)] i,j and [D dec (G(X))] i,j Represent the decision of the decoder at pixel point (i, j), ∑ i,j log[D dec (y)] i,j represents a first scalar value, used to indicate the probability that the discriminative encoder determines that the synthesized high-definition image belongs to the real high-definition image. i,j log[1-D dec (G(x))] i,j represents a third scalar value, which is used to indicate the probability that the corresponding position of each pixel point of the synthesized high-definition image determined by the discriminant decoder belongs to the high-definition image detail.

[0085] Furthermore, in the discriminator model structure, the encoder loss function is usually used to evaluate the discriminator's ability to discriminate the overall image or image features (i.e., structural discrimination). It is calculated based on the difference between the output of the discriminator encoder (i.e., the discrimination result) and the true label. This loss function guides the parameter update of the discriminator encoder part to improve its ability to distinguish between real and fake data. The decoder loss function is used to evaluate the discriminator's performance in restoring image details or performing pixel-level discrimination (i.e., detail discrimination).

[0086] It can be seen that in this example, the server introduces a discriminator into the preset generative adversarial network so that the discriminator can perform structural and detail discrimination on the enhanced image generated by the generator through data processing while generating the discrimination result, thereby guiding the generator to learn the structural and detail information in the real high-definition image, improve the generator's denoising and enhancement performance, and thus improve the stability of the trained generative adversarial network model.

[0087] In a possible embodiment, the discriminator further includes a skip link, and the discriminant encoder and the discriminant decoder are connected via a skip link, and the skip link is used to fuse the high-resolution features in the discriminant decoder with the low-resolution features in the network layer of the corresponding level in the discriminant encoder.

[0088] The design principle of the skip link is that the discriminative encoder part contains multiple convolutional layers and pooling layers, which are used to gradually extract and compress image features. The discriminative decoder part is the opposite process, which restores and reconstructs the compressed features. The skip link can directly connect the feature map of the shallower layer of the encoder (close to the input layer) to the deeper layer of the decoder (close to the output layer), so that information can be quickly transmitted between different layers.

[0089] Furthermore, the additionally designed skip links also include the following functions: 1. Feature fusion function: (1) Cross-layer feature integration: Skip links can achieve feature fusion across network layers. The discriminative encoder is responsible for extracting image features. As the network layer goes deeper, the features gradually change from high-resolution features containing more details to more abstract low-resolution semantic features. Skip links pass high-resolution detail features (such as object edges, textures, etc.) from earlier layers in the discriminative encoder to the discriminative decoder. (2) Combination of semantics and details: In the process of restoring image resolution, the discriminative decoder combines the detail features obtained from the skip links with its own semantic features, and can output feature maps that contain both rich details and accurate semantic understanding. Taking medical image segmentation as an example, the cell boundary details extracted by the early layers of the discriminative encoder are fused with the semantic features of the overall structure of the organ in the decoder through skip links, so that the segmentation result can accurately divide the cell boundaries and understand the organ region to which it belongs. (3) Utilization of multi-scale features: Feature maps of different layers represent image information at different scales, and skip links integrate this multi-scale information. When processing natural images, the model can simultaneously utilize local textures (shallower features) and overall contours (deeper features) to enhance the overall understanding of the image content. 2. Optimize the gradient propagation function: (1) Alleviate the gradient vanishing problem: During the training of deep neural networks, as the network depth increases, the problem of gradient vanishing or gradient diffusion may occur. Gradient vanishing means that during the backpropagation process, the gradient gradually approaches zero as the number of network layers increases, resulting in the inability to effectively update the weights of the previous layers. Skip links provide a direct path that allows gradients to be backpropagated from the decoder to the encoder more effectively. (2) Accelerate model convergence: Since skip links improve gradient propagation, the model can learn effective features faster, thereby accelerating convergence during training. This accelerated convergence effect is particularly evident in complex image tasks. For example, when processing high-resolution natural landscape image classification tasks, skip links can allow the model to learn the features of various objects in the landscape (such as trees and rivers) and the relationships between them more quickly, thereby achieving better classification results more quickly. 3. Improve model performance functions: (1) Improve segmentation accuracy (taking segmentation tasks as an example): By fusing feature information at different levels, jump links help improve the accuracy of image segmentation. In semantic segmentation and instance segmentation tasks, accurately identifying object boundaries and categories is the key. The information fusion brought by jump links enables the model to better grasp these key elements and output more accurate segmentation results. For example, when segmenting vehicles in urban street images, combining vehicle details (such as wheels, windows) and semantic information (such as the overall shape of the vehicle) can more accurately divide the area of ​​each vehicle. (2) Enhance robustness and generalization ability: Jump links enable the model to utilize features at multiple levels, which increases the model's adaptability to different types of images and different scenarios.For example, when encountering changes in image quality (such as noise interference, lighting changes, etc.) or changes in object morphology (such as object deformation, partial occlusion, etc.), the model can better cope with these changes by comprehensively utilizing features at different levels, thereby improving the robustness and generalization ability of the model.

[0090] Exemplarily, in addition to setting a skip link between the encoder and the decoder, the following possible improvements can be made: 1. Split the discriminative encoder and the discriminative decoder into multiple submodules, each of which contains an encoder and a decoder, and enhance the model's expressiveness and context-awareness through layer-by-layer upward and downward convolution feature fusion. Its advantage is that it can better capture multi-scale feature information in the image, reduce feature loss, and improve model accuracy. 2. Additional deep supervision technology is set, that is, technology used for deep neural network training. In deep neural networks, as the number of network layers increases, problems such as gradient vanishing or gradient explosion may occur, making the model difficult to train. The core idea of ​​deep supervision is to add additional supervision signals to the middle layer of the neural network to help the gradient propagate better, thereby accelerating the training process and potentially improving the performance of the model. Combine high-level semantics from feature maps of different scales with low-level semantics, and learn hierarchical representations from multi-scale aggregated feature maps. Its advantages are to improve model accuracy, reduce network parameters, and improve computational efficiency. 3. An attention gate module is added to the skip link and upsampling module between the discriminative encoder and the discriminative decoder. It is used to weight the feature information output by the discriminative encoder and the features of the corresponding layer in the discriminative decoder to generate a corresponding gating signal. The gate signal is used to control the discriminative encoder to distinguish the importance of features at different spatial positions in the enhanced image during image discrimination. The advantage is that by dynamically adjusting the weights, the network can pay more attention to the area of ​​interest, thereby improving the model accuracy.

[0091] It can be seen that in this example, based on the generator and discriminator preset by the above-mentioned generative adversarial network model, additional jump links can be set between the discriminative encoder and the discriminative decoder to improve segmentation accuracy, more accurately identify object boundaries and categories in image segmentation tasks, output more accurate results, and better distinguish different object instances in instance segmentation. In addition, the robustness and generalization capabilities are enhanced to adapt to different types of images and various scene changes, such as noise interference, lighting changes, object deformation or partial occlusion.

[0092] In a possible embodiment, the discriminator also includes an attention module, the output layer of the discriminant encoder and the output layer of the discriminant decoder are respectively connected to the input layer of the attention module, and the feature information includes a first output feature map; the feature information is input into the discriminant decoder to obtain a third scalar value and a fourth scalar value, including: inputting the first output feature map into the discriminant decoder to obtain a second output feature map; updating the size and number of channels of the first output feature map to the size and number of channels of the second output feature map through the attention module; and, determining the attention weight according to the updated first output feature map and the second output feature map through the attention module; obtaining a weighted second output feature map according to the attention weight and the second output feature map; and inputting the weighted second output feature map into the discriminant decoder to determine the third scalar value and the fourth scalar value.

[0093] Among them, the attention weight is used to characterize the relative importance of each pixel in the output feature map.

[0094] For example, see Figure 4 , Figure 4 Schematic diagram of another structure of generative adversarial network model training provided by the embodiment of the present application. Figure 4 As shown, on the basis of the above-mentioned discriminator model structure, an attention module is further provided in the discriminator, wherein the discriminant decoder is used to process detail discrimination of enhanced images and synthesized high-definition images, the discriminant encoder is used to process structural discrimination of enhanced images and synthesized high-definition images, and the output layer of the discriminant encoder and the output layer of the discriminant decoder are respectively connected to the input layer of the attention module. Since the preset jump link in the previous embodiment can fuse the first output feature map output by the discriminant encoder with the second output feature map output by the discriminant decoder, thereby reducing feature loss, the encoder learns the speckle noise information of the background part of the original image, and finally leads to a decrease in the denoising ability of the generator. Therefore, an attention module is introduced to discriminate the noise weight of the background part in the second output feature map output by the decoder, thereby improving the generator's ability to remove speckle noise. The newly added attention module has the following advantages: 1. Only one attention module is introduced to reduce model parameters and improve computational efficiency. 2. The attention module takes the final first output feature map of the discriminant encoder as input, so that the model pays more attention to structural information, reduces the weight of background noise, and improves model accuracy.

[0095] Furthermore, the first output feature map is first input into the discriminative decoder to obtain the second output feature map. Secondly, since the number of channels and size of the first output feature map of the discriminative encoder and the second output feature map of the discriminative decoder are different, convolution and superposition operations cannot be performed directly. Therefore, the size and number of channels of the first output feature map are updated to the size and number of channels of the second output feature map through the attention module.

[0096] For example, see Figure 5 , Figure 5 Schematic diagram of the structure of an attention module provided by an embodiment of the present application. Figure 5 As shown, the module structure of the attention module includes a two-dimensional convolution layer, a first convolution block, a first activation function, a second convolution block and a second activation function. Among them, the discriminant encoder will output a first output feature map, and the discriminant decoder will output a second output feature map. After that, the first output feature map output by the discriminant encoder will be sent to the two-dimensional convolution layer, and the two-dimensional convolution layer will adjust the first output feature map of the encoder part to be completely consistent with the second output feature map output by the discriminant decoder through a preset interpolation, and then use the first convolution block for subsequent processing. At the same time, the second feature map output by the discriminant decoder will be sent to the first convolution block for the same subsequent processing. After the processing is completed, the processed first output feature map is added to the processed second output feature map, and a nonlinear transformation is performed through the first activation function. This step is to fuse information and generate a feature map containing attention information. Then, the second convolution block and the second activation function are used to generate a normalized attention weight based on the feature map containing attention information, and the attention weight is used to characterize the relative importance of each pixel in the output feature map. Finally, the discriminant decoder performs discriminative processing by weighting the second output feature map to obtain the final third scalar value and fourth scalar value. The weighting mechanism of the attention module allows the discriminator to dynamically adjust the importance of its detail information according to its corresponding overall structural information when processing the detail information of each pixel position of the enhanced image and the synthesized high-definition image, so that the model can pay more attention to the details of the image.

[0097] The first convolutional block contains a 2D convolutional layer and a batch normalization layer. The number of input channels of the 2D convolutional layer is consistent with the number of output channels. The convolution kernel size is 1x1 and the step size is 1, which is used to adjust the number of channels of the feature map while keeping the spatial dimension unchanged. The number of features of the batch normalization layer is consistent with the number of output channels of the 2D convolutional layer, which is used to improve the generalization ability of the model. The first convolutional block contains a 2D convolutional layer and a batch normalization layer. The difference is that the number of output channels of convolutional block 2 is 1.

[0098] It can be seen that in this example, the server improves the model accuracy through the technical effect of the setting of the attention module, reduces the interference of irrelevant information on the model decision, better distinguishes the characteristics of objects of different categories in tasks such as image classification, and reduces the classification error rate. While ensuring the accuracy of the model, the computing efficiency is improved. A properly designed attention module can reduce model parameters, occupy less memory and computing resources, and speed up model training and reasoning.

[0099] Step S205, updating the parameters of the discriminator and the generator according to the loss function value to train the generative adversarial network model to obtain the target image enhancement model.

[0100] Among them, the target image enhancement model is a generative adversarial network model after training.

[0101] Among them, the trained target image enhancement model can improve the image quality of the original scanned image collected by the medical scanning device while removing noise, so that doctors can see the overall structure and image details of the target position of the test object more clearly. Figure 6a , Figure 6b , Figure 6c , Figure 6d , Figure 6e , Figure 6f .in, Figure 6a 1 is a schematic diagram of an OCT scan image provided by an embodiment of the present application. This is any one of a plurality of scan images acquired by a first medical scanning device for a target position of a detection object. It can be seen that the scan image contains a lot of speckle noise. Figure 6b This is a schematic diagram of a synthetic high-definition image provided by an embodiment of the present application. This is a synthetic high-definition image obtained by the server by repeatedly obtaining multiple scanned images through a scanner and then performing a high-definition image generation operation. It can be seen that the server adaptively enhances the local contrast of the image, while avoiding noise amplification, further improving the OCT image quality, enhancing the image contrast, making the lesion area clearer, and helping doctors diagnose. By looking at Figure 6c , Figure 6c is a schematic diagram of an enhanced image provided by an embodiment of the present application, Figure 6c is the target image enhancement model after training. Figure 6a The enhanced image obtained after the image enhancement operation is performed on the corresponding scanned image. It can be found that the enhanced image obtained by the image enhancement performed by the trained generator is the same as Figure 6b The high-definition synthetic images in are basically similar, indicating the effectiveness of the training method of the generative adversarial network model provided in this application, as well as the practicality and stability of the trained target image enhancement model for image enhancement.

[0102] Further, Figure 6d is a schematic diagram of an OCT lesion image provided in an embodiment of the present application, Figure 6e is an image schematic diagram of a synthetic high-definition image of an OCT lesion image provided in an embodiment of the present application, Figure 6f is a schematic diagram of an enhanced image of an OCT lesion image provided in an embodiment of the present application. Figure 6e Synthetic high-definition images for lesion images in Figure 6fFrom the enhanced images in the image, we can know that for different types of lesions in OCT images, the trained target image enhancement model can also accurately and practically denoise the scanned images and improve the image quality, thereby avoiding doctors from misjudging the patient's condition and improving the effectiveness and practicality of the training method.

[0103] Among them, the steps provided in this application are only represented as a single training process. In the actual training process, the server will repeatedly train the preset generative adversarial network model until the discrimination result obtained by the discriminator is less than the preset probability, and the first function value corresponding to the discriminator and the second function value corresponding to the generator fall into a preset smaller function value range. The smaller the loss function value, the closer the prediction result of the model is to the actual result. When the loss function value is small on the training data, and the model does not overfit the training data, then this model may also have a better performance on new, unseen data (test data). In the actual model training process, in order to avoid too many or too few model training times, when the generative adversarial network model training is completed, and the obtained discrimination result, the first function value and the second function value meet the requirements, the generative adversarial network model obtained by the final training can be determined as the target image enhancement model. The server can perform image enhancement on the scanned image collected by the medical scanning device in the actual medical clinical diagnosis to obtain an enhanced image close to the real high-definition image, thereby improving the stability of the generator and the practicality of image enhancement.

[0104] visible, Figure 2 This is a flow chart of a model training method based on image enhancement provided by an embodiment of the present application. The server superimposes a noise image on a scanned image and then inputs it into the generator for training and optimization to improve the stability and robustness of the model. The enhanced image and the synthesized high-definition image are distinguished by the discriminator to repeatedly train the model, thereby improving the image enhancement performance of the generator and the discrimination ability of the discriminator.

[0105] The following is an embodiment of the device of the present application. The embodiment of the device of the present application and the embodiment of the method of the present application are of the same concept and are used to execute the method described in the embodiment of the present application. For the convenience of explanation, the embodiment of the device of the present application only shows the part related to the embodiment of the device of the present application. For the specific technical details not disclosed, please refer to the description of the embodiment of the method of the present application, which will not be repeated here one by one.

[0106] The present invention provides a model training device based on image enhancement, which is applied to Figure 1The server 110 in the image enhancement system 100 shown in the figure, the image enhancement system 100 also includes a medical scanning device 120 and a display device 130, and the server 110 is respectively connected to the medical scanning device 120 and the display device 130 for communication. Specifically, the model training device based on image enhancement is used to execute the steps executed by the server in the above model training method based on image enhancement. The model training device based on image enhancement provided in the embodiment of the present application may include modules corresponding to the corresponding steps.

[0107] The embodiment of the present application can divide the functional modules of the model training device based on image enhancement according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. The division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0108] In the case of dividing each functional module into corresponding functional modules, Figure 7 is a functional unit block diagram of a model training device based on image enhancement provided in an embodiment of the present application; a model training device based on image enhancement 70 is applied to Figure 1 The server 110 in the image enhancement system 100 shown in the figure, the image enhancement system 100 also includes a medical scanning device 120 and a display device 130, and the server 110 is respectively connected to the medical scanning device 120 and the display device 130 for communication; the model training device 70 based on image enhancement includes: an image acquisition unit 701, used to acquire a scanned image and a noise image, the scanned image is a medical image acquired by scanning the target position of the detection object, and the noise image is a medical image acquired by scanning the background without the detection object; a preprocessing unit 702, used to perform a preprocessing operation on the scanned image according to the noise image, and determine a training image set; a first The training unit 703 is used to input the training images in the training image set into the generator in the preset generative adversarial network model to obtain the enhanced images corresponding to the training images; and, input the enhanced images and the synthetic high-definition images in the training image set into the discriminator in the generative adversarial network model to obtain the discrimination results and the loss function value corresponding to the discrimination results, and the discrimination results are used to indicate the probability that the discriminator determines that the enhanced image belongs to the real high-definition image; the second training unit 704 is used to update the parameters of the discriminator and the parameters of the generator according to the loss function value to train the generative adversarial network model to obtain the target image enhancement model, and the target image enhancement model is the generative adversarial network model after the training is completed.

[0109] In a possible embodiment, in terms of acquiring scanned images and noise images, the image acquisition unit 701 is specifically used to: scan the detection object through a first medical scanning device to acquire an initial acquisition image; determine the initial position of the imaging probe of the first medical scanning device in the horizontal direction, where the horizontal direction is a direction perpendicular to the light beam penetration direction corresponding to the imaging probe; control the first medical scanning device to repeatedly scan the detection object according to the initial position to acquire a preset number of subsequent acquisition images; integrate the initial acquisition image and the subsequent acquisition images to determine the scanned image; scan the background without the detection object in each acquisition mode through a second medical scanning device to acquire an aerial image, where the second medical scanning device includes one or more medical scanning devices of a different instrument type from that of the first medical scanning device; and integrate all the aerial images to determine the noise image.

[0110] In a possible embodiment, the training image set includes a plurality of reference training pairs, each of which includes a single training image and a single synthetic high-definition image. In terms of performing a preprocessing operation on the scanned image according to the noise image to determine the training image set, the preprocessing unit 702 is specifically used to: perform a high-definition image generation operation on the initial acquisition image and the subsequent acquisition image in the scanned image to generate a synthetic high-definition image to be processed for the target position; respectively combine the synthetic high-definition image to be processed and each initial acquisition image or the subsequent acquisition image in the scanned image into a plurality of initial training image pairs, and the scanned images in any two initial training image pairs are different from each other; perform a slider cropping operation on each initial training image pair according to a preset sliding step size and a sliding window size to generate a synthetic high-definition image to be processed for the target position; A plurality of sub-image pairs are obtained, and a slider cropping operation is performed on the noise image to obtain a plurality of noise image blocks, each sub-image pair includes a single scanned image block and a single high-definition image block; a single high-definition image block whose pixel average value is greater than a preset pixel threshold is determined as a processed synthetic high-definition image, and a sub-image pair including the processed synthetic high-definition image is determined as a reference training pair; noise superposition is performed on a single scanned image block and any noise image block in each reference training pair, and a simulation processing operation is performed on the scanned image block after noise superposition to obtain a training image in each reference training pair, and the simulation processing operation is used to perform image changes on the scanned image block after noise superposition to simulate the scanned image acquired by the first medical scanning device under different scanning environments.

[0111] In a possible embodiment, in performing a high-definition image generation operation on an initial acquisition image and subsequent acquisition images in a scanned image to generate a synthetic high-definition image to be processed for a target position, the preprocessing unit 702 is specifically used to: determine any subsequent acquisition image or the initial acquisition image in the scanned image as a reference image; perform a registration operation on the remaining scanned images except the reference image through affine transformation to align the reference image; determine multiple pixel averages according to the pixel value of each pixel position corresponding to the registered scanned image, each pixel average corresponds to a pixel position; generate an average image according to the pixel average corresponding to each pixel position; divide the average image into multiple reference image blocks of preset sizes, and calculate the grayscale histogram of each reference image block; perform a histogram equalization operation on each reference image block according to a preset contrast threshold and grayscale histogram to obtain a processed reference image block; and determine the synthetic high-definition image to be processed according to the processed reference image block.

[0112] In a possible embodiment, the discriminator includes a discriminative encoder and a discriminative decoder, and the loss function value includes a first function value and a second function value, the first function value is used to characterize the accuracy of the discriminator in distinguishing between the enhanced image and the synthesized high-definition image, and the second function value is used to characterize the error size between the enhanced image generated by the generator and the synthesized high-definition image; in terms of inputting the enhanced image and the synthesized high-definition image in the training image set into the discriminator in the generative adversarial network model to obtain the discrimination result and the loss function value corresponding to the discrimination result, the first training unit 703 is specifically used to: input the enhanced image and the synthesized high-definition image into the discriminative encoder to obtain feature information of the enhanced image and the synthesized high-definition image, a first scalar value and a second scalar value, the first scalar value is used to indicate the probability that the discriminative encoder determines that the synthesized high-definition image belongs to the real high-definition image, and the second scalar value is used to indicate the probability that the discriminative encoder determines that the enhanced image belongs to the real high-definition image; input the feature information into the discriminative decoder to obtain a third scalar value and a fourth scalar value, the third scalar value is used to indicate the probability that the discriminative decoder determines that the synthesized high-definition image is in The probability that the corresponding position of each pixel belongs to the high-definition image detail, and the fourth scalar value is used to indicate the probability that the enhanced image determined by the discriminant decoder at the corresponding position of each pixel belongs to the high-definition image detail; the discrimination result is determined according to the first scalar value, the second scalar value, the third scalar value and the fourth scalar value; the encoding loss value is determined according to the first scalar value, the second scalar value and the preset encoder loss function, and the decoding loss value is determined according to the third scalar value, the fourth scalar value and the preset decoder loss function, the encoding loss value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image and the synthesized high-definition image at the overall image level, and the decoding loss value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image and the synthesized high-definition image at the image detail level; the first function value is determined according to the preset discriminator loss function, the encoding loss value and the decoding loss value, the discriminator loss function includes a first variable coefficient corresponding to the first function value and a second variable coefficient corresponding to the second function value; and the second function value is determined according to the preset generator loss function, the second scalar value and the fourth scalar value.

[0113] In a possible embodiment, the discriminator further includes a skip link, and the discriminant encoder and the discriminant decoder are connected via a skip link, and the skip link is used to fuse the high-resolution features in the discriminant decoder with the low-resolution features in the network layer of the corresponding level in the discriminant encoder.

[0114] In a possible embodiment, the discriminator also includes an attention module, and the output layer of the discriminant encoder and the output layer of the discriminant decoder are respectively connected to the input layer of the attention module, and the feature information includes a first output feature map; in terms of inputting the feature information into the discriminant decoder to obtain the third scalar value and the fourth scalar value, the first training unit 703 is specifically used to: input the first output feature map into the discriminant decoder to obtain the second output feature map; through the attention module, update the size and number of channels of the first output feature map to the size and number of channels of the second output feature map; and, through the attention module, determine the attention weight according to the updated first output feature map and the second output feature map, the attention weight is used to characterize the relative importance of each pixel in the output feature map; according to the attention weight and the second output feature map, obtain a weighted second output feature map; input the weighted second output feature map into the discriminant decoder to determine the third scalar value and the fourth scalar value.

[0115] In the case of an integrated unit, such as Figure 8 As shown, Figure 8 1 is a functional unit block diagram of another model training device based on image enhancement provided by an embodiment of the present application. Figure 8 In the embodiment, the model training device 70 based on image enhancement includes: a processing module 802 and a communication module 801. The processing module 802 is used to control and manage the actions of the model training device based on image enhancement, for example, the steps of acquiring the image unit 701, the preprocessing unit 702, the first training unit 703 and the second training unit 704, and / or other processes for executing the technology described herein. The communication module 801 is used to support the interaction between the model training device based on image enhancement and other devices. Figure 8 As shown, the model training device based on image enhancement may include a storage module 803, and the storage module 803 is used to store the program code and data of the model training device based on image enhancement.

[0116] Among them, the processing module 802 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 801 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 803 can be a memory.

[0117] Among them, all relevant contents of each scene involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above image enhancement-based model training device 70 can execute the above Figure 2 The model training method based on image enhancement is shown.

[0118] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0119] Fig. 9 is a structural block diagram of a server provided in an embodiment of the present application. Fig. 9 As shown, the server 110 may include one or more of the following components: a processor 910, a memory 920 coupled to the processor 910, wherein the memory 920 may store one or more programs 921, and the one or more programs 921 may be configured to implement the methods described in the above embodiments when executed by one or more processors 910. The server here is the server 110 in the above embodiment, and the server may implement the model training method based on image enhancement described in the above embodiment.

[0120] The processor 910 may include one or more processing cores. The processor 910 uses various interfaces and lines to connect the various parts of the entire server 110, and executes various functions and processes data of the server 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 920, and calling data stored in the memory 920. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 910, but may be implemented separately through a communication chip.

[0121] The memory 920 may include a random access memory (RAM) or a read-only memory (ROM). The memory 920 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the server 110 during use, etc.

[0122] It is understandable that the server 110 may include more or fewer structural elements than those in the above structural block diagram, which is not limited here.

[0123] An embodiment of the present application also provides a computer storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, part or all of the steps of any method recorded in the above method embodiments are implemented.

[0124] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0125] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic; for example, the division of units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0129] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (directrambus RAM, DR RAM) and other media that can store program code.

[0130] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various changes and modifications, including the combination of the above-mentioned different functions and implementation steps, including software and hardware implementation methods, all of which are within the scope of protection of the present invention.

Claims

1. A model training method based on image enhancement, characterized in that: The method comprises: Acquire a scanned image and a noise image, wherein the scanned image is a medical image acquired by scanning a target position of a detection object, and the noise image is a medical image acquired by scanning a background where the detection object does not exist; According to the noise image, performing a preprocessing operation on the scanned image to determine a training image set; Inputting the training images in the training image set into the generator in the preset generative adversarial network model to obtain enhanced images corresponding to the training images; and Inputting the enhanced image and the synthetic high-definition image in the training image set into the discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result, wherein the discrimination result is used to indicate the probability that the discriminator determines that the enhanced image belongs to a real high-definition image; The parameters of the discriminator and the parameters of the generator are updated according to the loss function value to train the generative adversarial network model to obtain a target image enhancement model, wherein the target image enhancement model is the generative adversarial network model after training.

2. The method according to claim 1, characterized in that The step of acquiring the scanned image and the noise image comprises: Scanning the detection object by a first medical scanning device to obtain an initial acquisition image; Determining an initial position of an imaging probe of the first medical scanning device in a horizontal direction, wherein the horizontal direction is a direction perpendicular to a penetration direction of a light beam corresponding to the imaging probe; According to the initial position, controlling the first medical scanning device to repeatedly scan the detection object to obtain a preset number of subsequent acquisition images; Integrating the initial acquisition image and the subsequent acquisition image to determine the scanned image; Scanning the background without the detection object in each acquisition mode by a second medical scanning device to acquire an aerial image, wherein the second medical scanning device includes one or more medical scanning devices of different instrument types from those of the first medical scanning device; All of the aerial images are integrated to determine the noise image.

3. The method according to claim 2, characterized in that The training image set includes a plurality of reference training pairs, each reference training pair including a single training image and a single synthetic high-definition image; The step of performing a preprocessing operation on the scanned image according to the noise image to determine a training image set includes: Performing a high-definition image generation operation on the initial acquisition image and the subsequent acquisition image in the scanned image to generate a synthesized high-definition image to be processed for the target position; Combining each of the initial acquisition images or the subsequent acquisition images in the synthesized high-definition image to be processed and the scanned image into a plurality of initial training image pairs, respectively, wherein the scanned images in any two of the initial training image pairs are different from each other; According to a preset sliding step size and a sliding window size, a slider cropping operation is performed on each of the initial training image pairs to obtain a plurality of sub-image pairs, and the slider cropping operation is performed on the noise image to obtain a plurality of noise image blocks, each of the sub-image pairs including a single scanned image block and a single high-definition image block; Determine the single high-definition image block whose pixel average value is greater than a preset pixel threshold as the processed synthetic high-definition image, and determine the sub-image pair including the processed synthetic high-definition image as the reference training pair; Noise superposition is performed on the single scanned image block and any one of the noise image blocks in each of the reference training pairs, and a simulation processing operation is performed on the scanned image block after noise superposition to obtain the training image in each of the reference training pairs, wherein the simulation processing operation is used to perform image changes on the scanned image block after noise superposition to simulate the scanned images acquired by the first medical scanning device under different scanning environments.

4. The method according to claim 3, characterized in that The performing a high-definition image generation operation on the initial acquisition image and the subsequent acquisition image in the scanned image to generate a to-be-processed synthetic high-definition image for the target position includes: Determine any one of the subsequent collected images or the initial collected image in the scanned image as a reference image; Performing a registration operation on the remaining scanned images except the reference image through affine transformation to align the reference image; Determine a plurality of pixel averages according to the pixel value of each pixel position corresponding to the registered scanned image, each of the pixel averages corresponding to one pixel position; Generate an average image according to the pixel average value corresponding to each of the pixel positions; Dividing the average image into a plurality of reference image blocks of a preset size, and calculating a grayscale histogram of each of the reference image blocks; According to a preset contrast threshold and the grayscale histogram, a histogram equalization operation is performed on each of the reference image blocks to obtain a processed reference image block; The synthesized high-definition image to be processed is determined according to the processed reference image block.

5. The method according to any one of claims 1 to 4, characterized in that: The discriminator includes a discriminative encoder and a discriminative decoder, the loss function value includes a first function value and a second function value, the first function value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image from the synthesized high-definition image, and the second function value is used to characterize the error between the enhanced image generated by the generator and the synthesized high-definition image; The step of inputting the enhanced image and the synthetic high-definition image in the training image set into the discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result includes: Inputting the enhanced image and the synthesized high-definition image into the discriminative encoder to obtain feature information of the enhanced image and the synthesized high-definition image, a first scalar value and a second scalar value, wherein the first scalar value is used to indicate the probability that the discriminative encoder determines that the synthesized high-definition image belongs to the real high-definition image, and the second scalar value is used to indicate the probability that the discriminative encoder determines that the enhanced image belongs to the real high-definition image; Inputting the feature information into the discriminant decoder to obtain a third scalar value and a fourth scalar value, wherein the third scalar value is used to indicate the probability that the corresponding position of each pixel of the synthesized high-definition image determined by the discriminant decoder belongs to the high-definition image detail, and the fourth scalar value is used to indicate the probability that the corresponding position of each pixel of the enhanced image determined by the discriminant decoder belongs to the high-definition image detail; determining the discrimination result according to the first scalar value, the second scalar value, the third scalar value and the fourth scalar value; Determine an encoding loss value according to the first scalar value, the second scalar value and a preset encoder loss function, and determine a decoding loss value according to the third scalar value, the fourth scalar value and a preset decoder loss function, wherein the encoding loss value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image and the synthesized high-definition image at the overall image level, and the decoding loss value is used to characterize the accuracy of the discriminator in distinguishing the enhanced image and the synthesized high-definition image at the image detail level; The first function value is determined according to a preset discriminator loss function, the encoding loss value and the decoding loss value, wherein the discriminator loss function includes a first variable coefficient corresponding to the first function value and a second variable coefficient corresponding to the second function value; and the second function value is determined according to a preset generator loss function, the second scalar value and the fourth scalar value.

6. The method according to claim 5, characterized in that The discriminator also includes a skip link, and the discriminant encoder and the discriminant decoder are connected via a skip link, and the skip link is used to fuse the high-resolution features in the discriminant decoder with the low-resolution features in the network layer of the corresponding level in the discriminant encoder.

7. The method according to claim 5, characterized in that The discriminator further includes an attention module, the output layer of the discriminant encoder and the output layer of the discriminant decoder are respectively connected to the input layer of the attention module, and the feature information includes a first output feature map; The inputting the feature information into the discriminant decoder to obtain a third scalar value and a fourth scalar value comprises: Inputting the first output feature map into the discriminative decoder to obtain a second output feature map; By means of the attention module, the size and the number of channels of the first output feature map are updated to the size and the number of channels of the second output feature map; and Determining, by the attention module, an attention weight according to the updated first output feature map and the second output feature map, wherein the attention weight is used to characterize the relative importance of each pixel in the output feature map; Obtaining a weighted second output feature map according to the attention weight and the second output feature map; The weighted second output feature map is input to the discriminative decoder to determine the third scalar value and the fourth scalar value.

8. A model training device based on image enhancement, characterized in that: The device comprises: An image acquisition unit, used to acquire a scanned image and a noise image, wherein the scanned image is a medical image acquired by scanning a target position of a detection object, and the noise image is a medical image acquired by scanning a background where the detection object does not exist; A preprocessing unit, configured to perform a preprocessing operation on the scanned image according to the noise image to determine a training image set; A first training unit is configured to input a training image in the training image set into a generator in a preset generative adversarial network model to obtain an enhanced image corresponding to the training image; and input the enhanced image and a synthesized high-definition image in the training image set into a discriminator in the generative adversarial network model to obtain a discrimination result and a loss function value corresponding to the discrimination result, wherein the discrimination result is used to indicate the probability that the discriminator determines that the enhanced image belongs to a real high-definition image; The second training unit is used to update the parameters of the discriminator and the parameters of the generator according to the loss function value to train the generative adversarial network model to obtain a target image enhancement model, wherein the target image enhancement model is the generative adversarial network model after training.

9. A server, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images