Image generation method

Abstract

The embodiment of the invention provides an image generation method. Acquiring an eye fundus image of the eye part acquired by the eye fundus camera; processing a gray value in the fundus image based on the saturation threshold and the linear mapping function to obtain a to-be-used image corresponding to the fundus image; inputting the image to be used into a pre-trained target processing model, and outputting an inner-layer face image and an outer-layer face image; performing pixel point fusion on the inner-layer face image and the outer-layer face image based on a target fusion function to obtain a full-layer face image of the eye fundus image; and displaying the full-layer face image. According to the technical scheme provided by the embodiment of the invention, the two-dimensional image acquired by the fundus camera can be used for generating the inner-layer and outer-layer face images of the OCT based on the neural network algorithm; and generating a full-layer face image based on the target fusion function. According to the technical scheme, deeper-level structural feature information of the eyes is obtained, a user can better understand all components and functions of the eyes, and eye abnormity can be found in the early stage.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to artificial intelligence technology, and more particularly to an image generation method, device, electronic device, and storage medium. Background Art

[0002] With the development of science and technology, fundus cameras and optical coherence tomography (OCT), as two common imaging instruments for ophthalmic examinations, are widely used in the field of medical imaging. The two different ophthalmic examination instruments each have their own unique advantages and applicable scenarios, and each has its own unique advantages in the presentation of ophthalmic images.

[0003] Currently, if workers want to obtain images taken by fundus cameras and enface images of OCT, they need to use fundus cameras and OCT devices to collect images of the eyes respectively. When using OCT equipment for examination, workers need to drop mydriatic drops into the user's eyes, which may cause discomfort in the eyes. In addition, using OCT equipment to examine the eye condition is not only cumbersome for workers, but also expensive for users, which is not conducive to saving medical resources. Summary of the invention

[0004] The disclosed embodiments provide an image generation method, device, electronic device and storage medium, which generate inner and outer enface images of OCT based on two-dimensional images collected by a fundus camera, and generate full-layer enface images based on a target fusion function. The technical solution of the disclosed embodiments generates an enface image of OCT without using OCT, and obtains deeper structural feature information of the eye, which helps users better understand the various components of the eye and their functions, and helps to detect eye abnormalities at an early stage.

[0005] In a first aspect, an embodiment of the present disclosure provides an image generation method, the method comprising:

[0006] Acquire a fundus image of an eye part collected by a fundus camera, wherein the fundus image is a grayscale image;

[0007] Based on a preset saturation threshold and a linear mapping function, grayscale values ​​in the fundus image are processed to obtain an image to be used corresponding to the fundus image;

[0008] Inputting the image to be used into a pre-trained target processing model, and outputting an inner enface image and an outer enface image of the eye area;

[0009] Perform pixel fusion on the inner enface image and the outer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image;

[0010] The full-thickness enface image is displayed.

[0011] In a second aspect, the present disclosure also provides an image generating device, the device comprising:

[0012] An image acquisition module, used to acquire a fundus image collected by a fundus camera of an eye part, wherein the fundus image is a grayscale image;

[0013] An image grayscale value processing module, used to process the grayscale value in the fundus image based on a preset saturation threshold and a linear mapping function to obtain an image to be used corresponding to the fundus image;

[0014] An inner and outer enface image generation module, used for inputting the image to be used into a pre-trained target processing model, and outputting an inner enface image and an outer enface image of the eye area;

[0015] A full-layer enface image generation module, used for performing pixel fusion on the inner layer enface image and the outer layer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image;

[0016] The full-layer enface image display module is used to display the full-layer enface image.

[0017] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0018] one or more processors;

[0019] a storage device for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generating method as described in any one of the embodiments of the present disclosure.

[0021] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image generation method as described in any one of the embodiments of the present disclosure.

[0022] The technical solution of the embodiment of the present disclosure obtains a fundus image collected by a fundus camera of an eye part, wherein the fundus image is a grayscale image, and then, based on a preset saturation threshold and a linear mapping function, the grayscale value in the fundus image is processed to obtain an image to be used corresponding to the fundus image, so that the image to be used is input into a pre-trained target processing model, and the inner enface image and the outer enface image of the eye part are output, and further, the inner enface image and the outer enface image are pixel-fused based on a target fusion function to obtain a full-layer enface image of the fundus image, and finally, the full-layer enface image is displayed. The technical solution of the embodiment of the present disclosure, the technical solution of the embodiment of the present disclosure, based on a neural network algorithm, generates an inner and outer enface image of OCT from a two-dimensional image collected by a fundus camera; and based on a target fusion function, generates a full-layer enface image. The technical solution of the embodiment of the present disclosure generates an OCT enface image without using an OCT device, and obtains deeper structural feature information of the eye, which helps users better understand the various components and functions of the eye, and helps to detect eye abnormalities at an early stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0024] Figure 1 is a flowchart of an image generation method provided by an embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of image generation to be used provided by an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of generating an inner layer enface image provided by an embodiment of the present disclosure;

[0027] Figure 4 is a schematic diagram of generating an outer enface image provided by an embodiment of the present disclosure;

[0028] Figure 5 is a schematic diagram of a process for determining a target training sample provided by an embodiment of the present disclosure;

[0029] Figure 6 is a schematic diagram of model parameter correction of a target processing model provided by an embodiment of the present disclosure;

[0030] Figure 7is a structural schematic diagram of an image generating device provided by an embodiment of the present disclosure;

[0031] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0033] Before introducing the embodiments of the present disclosure, the application scenarios of the embodiments of the present invention may be described first. When the staff uses a fundus camera to collect eye images of the user, only two-dimensional images based on the fundus camera are collected, lacking high-resolution deeper structural feature information of the eye. In this case, the fundus images collected by the fundus camera can be processed based on the technical method provided in the embodiments of the present invention to generate inner enface images and outer enface images; and based on the target fusion function, a full-layer enface image is generated. Furthermore, obtaining deeper structural feature information of the eye can help users better understand the various components of the eye and their functions, and help to detect eye abnormalities at an early stage.

[0034] Embodiment 1

[0035] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings. Figure 1 This is a flow chart of an image generation method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where a fundus image acquired by a fundus camera is processed using a target processing model to obtain inner enface and outer enface images and generate a full-layer enface image. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server. The technical solution provided by the embodiment of the present disclosure is executed based on the server.

[0036] like Figure 1 As shown, the method includes:

[0037] S110: Acquire a fundus image of the eye captured by a fundus camera.

[0038] Among them, the fundus camera is an instrument that uses non-contact intraocular photography to examine the fundus. The fundus camera is one of the important means for workers to observe the morphology, distribution and functional status of retinal blood vessels during work.

[0039] Among them, fundus images refer to fundus photos taken by medical devices such as fundus cameras. The fundus usually refers to the inside of the back of the eyeball, which is composed of the optic disc, macula, retina and some blood vessels, and appears orange-red.

[0040] It should be noted that the fundus image may be an image received by the server and captured by the user in real time using a fundus camera, or may be a stored image retrieved by the server from a related database.

[0041] It should be noted that in a specific application scenario, the fundus image can be acquired in real time, or when it is detected that the user takes a fundus image, the image can be acquired and used as the fundus image. The embodiments of the present disclosure do not specifically limit this.

[0042] S120 , based on a preset saturation threshold and a linear mapping function, the grayscale value in the fundus image is processed to obtain an image to be used corresponding to the fundus image.

[0043] Among them, since the fundus images taken by the fundus camera are all grayscale images, each pixel of the collected grayscale image corresponds to its own grayscale value. The grayscale values ​​of all pixels in the fundus image are between 0-255.

[0044] Among them, the saturation threshold is used to limit the grayscale value of the grayscale image within the range of upper and lower thresholds. The saturation threshold can be set to a preset threshold, which can be set to a first threshold and a second threshold. Optionally, the first threshold can be a lower threshold of the saturation threshold, and the second threshold can be an upper threshold of the saturation threshold. The grayscale values ​​of the pixels of the grayscale image below the lower threshold are set to the lower threshold, and the grayscale values ​​of the pixels of the grayscale image above the upper threshold are set to the upper threshold. For example: assuming that the lower threshold is 30 and the upper threshold is 240, then the grayscale values ​​of all pixels in the grayscale image whose grayscale values ​​are lower than 30 are set to 30, and the grayscale values ​​of all pixels in the grayscale image whose grayscale values ​​are higher than 240 are set to 240.

[0045] The linear mapping function refers to a function for normalizing the grayscale values ​​of pixels in the image after saturation threshold processing.

[0046] The image to be used is an image corresponding to the fundus image obtained by processing the grayscale value in the fundus image based on a preset saturation threshold and a linear mapping function.

[0047] Specifically, based on the fundus image captured by the fundus camera, the image is firstly thresholded according to the upper threshold and the lower threshold, and then the image is normalized. The image after gray value processing is the image to be used.

[0048] Optionally, based on a pre-set saturation threshold and a linear mapping function, the grayscale values ​​in the fundus image are processed to obtain an image to be used corresponding to the fundus image, including: determining grayscale value statistical information according to the grayscale values ​​of all pixels in the fundus image; determining a first critical grayscale value and a second critical grayscale value according to a first threshold and a second threshold in the saturation threshold; processing the grayscale values ​​of the fundus image according to the first critical grayscale value and the second critical grayscale value to obtain a pixel image to be mapped; processing the grayscale values ​​in the pixel image to be mapped based on a linear mapping function to obtain the image to be used.

[0049] Among them, the gray value statistics refer to the statistics of the gray value data of all pixels in the fundus image. First, the number of pixels accumulated to each gray value in the gray value of 0-255 in the gray image is calculated. Then the percentage of the number of pixels accumulated to each gray value to all pixels is the gray value statistics.

[0050] Among them, the grayscale value statistical information of the fundus image is determined by using the grayscale cumulative statistical histogram, see Figure 2 . For example, assuming that the number of pixels in the fundus photograph taken is M*N, the grayscale values ​​of all M*N pixels are accumulated in grayscale to obtain a grayscale statistical histogram. The horizontal axis of the grayscale statistical histogram represents the grayscale value, and the grayscale value range is 0-255 with a step size of 1. The vertical axis represents the number of times the pixel appears in the grayscale image at each grayscale value. The number of times the pixel appears in the grayscale image at each grayscale value is recorded as counts(i), where i is 0-255, and then counts(i) is divided by all pixels M*N to calculate the proportion of the number accumulated at each grayscale value, counts_ratio(i). The specific formula is: counts_ratio(i) = counts(i) / (M*N). After obtaining the proportion of the number accumulated at each grayscale value, subsequent calculations are performed.

[0051] The first threshold and the second threshold are pre-set thresholds that can be used to screen the number of proportions. The first threshold can be the minimum value of the proportion, and the second threshold can be the maximum value of the proportion. The first threshold and the second threshold are both between 0 and 1.

[0052] The first threshold is used to determine the first critical grayscale value, and the second threshold is used to determine the second critical grayscale value. The first critical grayscale value refers to the i value corresponding to the number of grayscale values ​​when the proportion is closest to the first threshold; the second critical grayscale value refers to the i value corresponding to the number of grayscale values ​​when the proportion is closest to the second threshold.

[0053] For example, see Figure 2 , the first threshold is a preset threshold for screening the first critical grayscale value, and optionally, the first threshold is 1%, that is, the proportion corresponding to each cumulative grayscale value is determined in order from low to high according to the cumulative grayscale value, and the grayscale value corresponding to the total cumulative proportion of 3% is used as the first critical grayscale value. The second threshold is a preset threshold for screening the second critical grayscale value, and optionally, the first threshold is 99.7%, that is, the proportion corresponding to each cumulative grayscale value is determined in order from low to high according to the cumulative grayscale value, and the grayscale value corresponding to the total cumulative proportion of 99.7% is used as the second critical grayscale value. In the embodiment of the present disclosure, for the pixel point in the fundus image, when the grayscale value of the pixel point is less than the first critical grayscale value, the grayscale value is adjusted to the first critical grayscale value; when the grayscale value of the pixel point is greater than the second critical grayscale value, the grayscale value is adjusted to the second critical grayscale value.

[0054] For example, assuming that the first critical grayscale value min is calculated to be 10, then the grayscale values ​​of the pixels between 0 and 10 are all set to grayscale value 10; assuming that the second critical grayscale value max is calculated to be 245, then the grayscale values ​​of the pixels between 245 and 255 are all set to grayscale value 245.

[0055] In the embodiment of the present disclosure, for a pixel point in the pixel image to be mapped, the grayscale value of the pixel point, the second critical grayscale value and the first critical grayscale value are substituted into the linear mapping function to obtain the mapping grayscale value of the pixel point; based on the mapping grayscale value of each pixel point in the pixel image to be mapped, the image to be used is determined.

[0056] Among them, the linear mapping function is a function that normalizes the pixel image to be mapped. The linear mapping function is defined as follows: f(x:min,max)=(x-min) / (max-min), where x is the gray value of the pixel, min is the first critical gray value, and max is the second critical gray value. Through the calculation of the linear mapping function, the pixel points in the pixel image to be mapped are converted from the min-max value to the 0-1 value.

[0057] Continue to see Figure 2 , after transformation by the linear mapping function, the mapping gray value of each pixel is obtained, and then the image to be used is determined.

[0058] The benefits of normalization are: normalizing images can improve the robustness of images, accelerate the training process, improve the accuracy of models, enhance the visual effects of images, and reduce numerical instability.

[0059] S130, inputting the image to be used into the pre-trained target processing model, and outputting the inner enface image and the outer enface image of the eye area.

[0060] The image to be used refers to the fundus grayscale image processed by the saturation threshold and the linear mapping function.

[0061] Among them, the target processing model includes a main network model and a secondary network model. The main network model is used to generate an inner enface image from a fundus image; the network structure of the secondary network model, the secondary network model is used to generate an outer enface image from a fundus image.

[0062] Among them, OCT is a new fundus imaging device. OCT is based on the existence of blood cells flowing in the fundus blood vessels, and performs repeated coherent optical tomography on the same cross section. OCT obtains the signal of moving blood cells, that is, blood flow, through a special calculation method, and reconstructs the vascular structure in three dimensions based on it. Enface image refers to the image of fundus blood vessels presented layer by layer in the form of coronal plane.

[0063] Among them, the inner layer is defined as from the inner limiting membrane (ILM) of the retinal structure to the inner nuclear layer (INL), and the inner enface image refers to the two-dimensional image of the fundus blood vessels from the ILM to the INL of the retinal structure; the outer layer is defined as from the outer limiting membrane (ONL) of the retinal structure to the pigment epithelium (RPE), and the outer enface image refers to the two-dimensional image of the fundus blood vessels from the ONL to the RPE of the retinal structure.

[0064] Among them, the target processing model uses a deep learning algorithm, which refers to a model that outputs the corresponding inner enface image and outer enface image after inputting the fundus grayscale image taken by the fundus camera.

[0065] Optionally, the image to be used is processed respectively based on the main network model and the secondary network model in the target processing model to obtain an inner enface image output by the main network model and an outer enface image output by the secondary network model.

[0066] Specifically, for the image to be used, the inner enface image is output according to the main network model.

[0067] For example, when the main network model is used to process images, it is recommended to consist of the following four components: 3x3 convolution with a span of 2, 1x1 convolution, dimension splicing and deformation unit. The 3x3 convolution with a span of 2 is mainly for effective feature convolution and appropriate downsampling. The 1x1 convolution actually performs a fully connected operation on the channel of each pixel, and this fully connected layer is trained for each pixel separately in order to better express and transform features. Dimension splicing is mainly to connect the input tensor sequence along a new dimension. The deformation unit is mainly for interpolation transformation of the image size.

[0068] Among them, see Figure 3 , the sub-network model network is mainly composed of an encoder and a decoder.

[0069] The task of the encoder is to convert the input data into a mathematical form that can be processed by the neural network. The main function of the encoder is to use the convolution operator, downsampling unit and activation function in deep learning to encrypt the outer features and generate a high-dimensional space that can express the outer details; the task of the decoder is to solve the mathematical problem and convert it into a specific solution. The decoder is mainly composed of upsampling and dimension splicing in deep learning. The main function is to decrypt the high-dimensional space generated by the encoder and express the transformation towards the target outer enface image feature space. The unsupervised algorithm of the common model framework of the encoder-decoder is used for the encoding-decoding structure design. A machine learning method that discovers the internal structure and pattern of data by learning the characteristics of the data itself without pre-labeled data labels.

[0070] S140 , performing pixel fusion on the inner enface image and the outer enface image based on the target fusion function to obtain a full-layer enface image of the fundus image.

[0071] Among them, the range of the full layer is the entire range from the inner layer to the outer layer, that is, from the inner limiting membrane to the pigment epithelial cell layer of the retinal structure.

[0072] Among them, the target fusion function is a piecewise function that fuses the inner layer enface image and the outer layer enface image to generate a full layer enface image.

[0073] Optionally, the target fusion function is:

[0074]

[0075] Among them, I1 (x, y) represents the gray value of the pixel at coordinate (x, y) in the inner enface image. 2 (x, y) represents the grayscale value of the pixel at the coordinate (x, y) in the outer enface image, and the range of the value is the height and width of the inner enface image and the outer enface image. w is a weight factor with a range between 0 and 1, which can be set by the user. M and N are positive odd numbers preset by the user, u and v are temporary variables that rotate the value, u ranges from -(M+1) / 2 to (M+1) / 2, and v ranges from -(N+1) / 2 to (N+1) / 2.

[0076] In the embodiment of the present disclosure, the inner layer enface image and the outer layer enface image are pixel-fused based on the target fusion function to obtain a full-layer enface image of the fundus image, including:

[0077] For the pixel point at the same position, the target piecewise function of the pixel point in the target fusion function is determined according to the pixel value of the pixel point in the inner enface image and the outer enface image respectively;

[0078] Substitute the pixel value of the pixel point into the target piecewise function to determine the display grayscale value of the pixel point.

[0079] Exemplarily, as a preferred solution, w is 0.7, and M and N are 7. When the grayscale value at the pixel point with coordinate (x, y) in the inner enface image is greater than or equal to the grayscale value at the pixel point with coordinate (x, y) in the outer enface image, the grayscale value at the pixel point with coordinate (x, y) in the full enface image is calculated as follows: That is, the grayscale value of the pixel at the coordinate (x, y) of the full-layer enface image is obtained; when the grayscale value of the pixel at the coordinate (x, y) in the inner layer enface image is less than the grayscale value of the pixel at the coordinate (x, y) in the outer layer enface image, the calculation formula of the grayscale value of the pixel at the coordinate (x, y) in the full-layer enface image is That is, first obtain the grayscale values ​​of the pixels around the coordinate (x, y) in the inner enface image, where the x and y values ​​rotate from -4 to 4, then obtain the grayscale values ​​of the pixels around the coordinate (x, y) in the outer enface image, where the x and y values ​​rotate from -4 to 4, then calculate the cross product values ​​of the corresponding pixels of the inner enface image and the outer enface image, then sum up all the cross product values, and finally divide the sum of the cross product values ​​by M*N to obtain the grayscale value of the pixel at the coordinate (x, y) in the full-layer enface image.

[0080] S150, display the full-layer enface image.

[0081] Among them, based on the displayed grayscale values ​​of all pixels, a full-layer enface image is determined after the inner-layer enface image and the outer-layer enface image are fused, and then the full-layer enface image is displayed.

[0082] The technical solution of the disclosed embodiment is to obtain a fundus image collected by a fundus camera of the eye part, and then process the grayscale value in the fundus image based on a preset saturation threshold and a linear mapping function to obtain an image to be used corresponding to the fundus image, and further input the image to be used into a pre-trained target processing model to output an inner enface image and an outer enface image of the eye part, and further perform pixel fusion on the inner enface image and the outer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image, and finally display the full-layer enface image. Without using OCT, deeper structural feature information of the eye is obtained, which helps users better understand the various components and functions of the eye and helps to detect eye abnormalities at an early stage.

[0083] Embodiment 2

[0084] Figure 5 It is a flow chart of determining a target training sample provided by an embodiment of the present disclosure. On the basis of the aforementioned embodiment, the target training sample can also be determined, and then the model parameters of the target processing model can be corrected. The correction method includes: correcting the model parameters according to the training input and the training output to determine the model parameters of the target processing model to meet the convergence conditions; performing loss processing on the fused predicted full-layer enface image and the test full-layer enface image, and correcting the model parameters of the target processing model based on the loss value. The specific implementation method can be found in the technical solution of this embodiment. Among them, the technical terms that are the same as or corresponding to the above embodiments are not repeated here.

[0085] like Figure 5 As shown, the method specifically comprises the following steps:

[0086] S210, obtaining a plurality of historical samples to be processed, where the historical samples to be processed include historical fundus images, historical inner enface images, historical outer enface images, and historical full-layer enface images at the same acquisition angle and location.

[0087] The historical samples to be processed include historical fundus images acquired by fundus cameras, and those acquired by OCT include historical inner enface images, historical outer enface images, and historical full-layer enface images of the same acquisition angle and location.

[0088] The historical samples to be processed may be images taken by a fundus camera or OCT, or images pre-stored in a storage space.

[0089] Specifically, before processing the historical samples, it is necessary to obtain multiple historical samples to train the model based on the samples. In order to improve the accuracy of the model, as many and rich historical samples as possible can be obtained, and multiple historical sample images can be obtained.

[0090] S220 , processing the above four images respectively based on a preset saturation threshold and a linear mapping function to obtain training images corresponding to the above four images respectively.

[0091] It should be noted that, for each historical sample, the method of S220 may be used to process it, so as to obtain a corresponding training image.

[0092] Specifically, based on the pre-set saturation threshold and the linear mapping function, the historical fundus images, historical inner layer enface images, historical outer layer enface images and historical full-layer enface images are processed respectively to obtain a training fundus image corresponding to the historical fundus image, a training inner layer enface image corresponding to the historical inner layer enface image, a training outer layer enface image corresponding to the historical outer layer enface image, and a training full-layer enface image corresponding to the historical full-layer enface image.

[0093] S230: Determine a target training sample based on the training image corresponding to each historical sample to be processed.

[0094] The training images include training fundus images, training inner enface images, training outer enface images, and training full-layer enface images.

[0095] The target training samples refer to the samples used for training in the target processing model.

[0096] Optionally, after obtaining the target training sample, for the target training sample, the training fundus image in the target training sample is input into the processing model to be trained, the training inner enface image and the training outer enface image are used as the output of the processing model to be trained, and the model parameters in the processing model to be trained are corrected; the model parameters corresponding to the time when the processing model to be trained reaches the convergence conditions are used as the model parameters of the target processing model.

[0097] It should be noted that when building a model and conducting experimental tests, it is sometimes found that the model has systematic deviations, indicating that the model is defective and therefore cannot maintain equivalence. In this case, the model needs to be corrected.

[0098] It should also be noted that a target processing model should have a certain output when the input amount remains unchanged. Reaching the convergence condition generally means that there is no large fluctuation in the training and validation loss curves, and as the number of training rounds increases, the fluctuation can still be within a certain tolerance range. Reaching the convergence condition means that the system is stable. When a certain weight parameter of the model changes slightly, the model output result will not change drastically and will not cause the system to crash. At this time, the model is fully trained and stable.

[0099] Specifically, the model parameters corresponding to when the convergence condition is reached are used as the model parameters of the target processing model.

[0100] Optionally, a test fundus image in the test sample is input into a target processing model, and a test inner layer enface image and a test outer layer enface image corresponding to the test fundus image are output; wherein the test sample includes at least a target training sample;

[0101] After fusing the test inner layer enface image and the test outer layer enface image based on the target fusion function, the predicted full layer enface image is obtained;

[0102] Perform loss processing on the predicted full-layer enface image and the test full-layer enface image in the test sample, and update the model parameters in the target processing model based on the loss value;

[0103] The loss processing includes determining the gray value loss processing of pixels in the predicted full-layer enface image and the tested full-layer enface image.

[0104] The test fundus image refers to a fundus image collected or stored using a fundus camera and for which an inner enface image and an outer enface image need to be generated.

[0105] The test inner layer enface image and the test inner layer enface image refer to images output after the test fundus image is processed by the target processing model.

[0106] The predicted full-layer enface image is the full-layer enface image obtained by fusing the test inner-layer enface image and the test outer-layer enface image.

[0107] It should be noted that loss processing refers to comparing the grayscale values ​​of each pixel corresponding to the predicted full-layer enface image and the test full-layer enface image. The proportion of corresponding pixels with different grayscale values ​​to all pixels is called the loss value. According to the loss value, the model parameters of the target processing model are updated.

[0108] The technical solution of the disclosed embodiment is to obtain multiple historical samples to be processed, which include historical fundus images, historical inner layer enface images, historical outer layer enface images and historical full-layer enface images of the same acquisition angle and position, and then, based on a preset saturation threshold and a linear mapping function, the historical fundus images, historical inner layer enface images, historical outer layer enface images and historical full-layer enface images are processed respectively to obtain training fundus images corresponding to the historical fundus images, training inner layer enface images corresponding to the historical inner layer enface images, training outer layer enface images corresponding to the historical outer layer enface images, and training full-layer enface images corresponding to the historical full-layer enface images; finally, based on the training fundus images, training inner layer enface images, training outer layer enface images and training full-layer enface images corresponding to each historical sample to be processed, the target training sample is determined. In addition, the model parameters of the target processing model are corrected to make the model fully trained and stable, thereby making up for the lack of deep projection information and providing staff with more sufficient eye information, which helps users better understand the various components of the eye and their functions and helps to detect eye abnormalities at an early stage.

[0109] Embodiment 3

[0110] Figure 7 is a structural schematic diagram of an image generating device provided by an embodiment of the present disclosure, such as Figure 7 As shown, the device includes: an image acquisition module 310, a to-be-used image generation module 320, an inner layer and outer layer enface image generation module 330, a full layer enface image generation module 340 and a full layer enface image display module 350.

[0111] An image acquisition module 310 is configured to acquire a fundus image of an eye portion acquired by a fundus camera, wherein the fundus image is a grayscale image;

[0112] The image to be used generating module 320 processes the grayscale value in the fundus image based on a preset saturation threshold and a linear mapping function to obtain an image to be used corresponding to the fundus image;

[0113] The inner and outer enface image generation module 330 inputs the image to be used into a pre-trained target processing model, and outputs the inner enface image and the outer enface image of the eye area;

[0114] A full-layer enface image generation module 340 performs pixel fusion on the inner layer enface image and the outer layer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image;

[0115] The full-layer enface image display module 350 displays the full-layer enface image.

[0116] On the basis of the above technical solutions, the image generation module 320 is further used to process the grayscale value in the fundus image based on a preset saturation threshold and a linear mapping function to obtain an image to be used corresponding to the fundus image.

[0117] Based on the above technical solutions, the image generation module 320 includes a gray value information statistics submodule, a first and a second critical gray value determination submodule, a pixel to-be-mapped image generation submodule and a to-be-used image generation submodule.

[0118] A grayscale value information statistics submodule, which determines grayscale value statistics information according to the grayscale values ​​of all pixels in the fundus image;

[0119] A first and a second critical grayscale value determination submodule, which determines a first critical grayscale value and a second critical grayscale value according to a first threshold value and a second threshold value in the saturation threshold value;

[0120] A pixel image to be mapped generating submodule processes the grayscale value of the fundus image according to the first critical grayscale value and the second critical grayscale value to obtain a pixel image to be mapped;

[0121] The image to be used generating submodule processes the grayscale values ​​in the pixel to be mapped image based on a linear mapping function to obtain the image to be used.

[0122] On the basis of the above-mentioned technical solutions, the pixel to-be-mapped image generation submodule also adjusts the grayscale value of a pixel point in the fundus image to the first critical grayscale value when the grayscale value of the pixel point is less than the first critical grayscale value; and adjusts the grayscale value to the second critical grayscale value when the grayscale value of the pixel point is greater than the second critical grayscale value.

[0123] On the basis of the above-mentioned technical solutions, the submodule for generating the image to be used also substitutes the grayscale value of the pixel point, the second critical grayscale value and the first critical grayscale value into the linear mapping function for the pixel point in the pixel image to be mapped, so as to obtain the mapping grayscale value of the pixel point; and determines the image to be used according to the mapping grayscale value of each pixel point in the pixel image to be mapped.

[0124] On the basis of the above-mentioned technical solutions, the inner and outer enface image generation module 330 also processes the image to be used based on the main network model and the secondary network model in the target processing model, so as to obtain the inner enface image output by the main network model and the outer enface image output by the secondary network model.

[0125] Based on the above technical solutions, the full-layer enface image generation module 340 has a target fusion function as follows:

[0126] Among them, I 1 (x, y) represents the gray value of the pixel at coordinate (x, y) in the inner enface image. 2 (x, y) represents the gray value of the pixel at the coordinate (x, y) in the outer enface image, w is a weight factor ranging between 0 and 1, M and N are preset positive odd numbers, u and v are temporary variables that take values ​​in rotation, u ranges from integers of -M / 2 to M / 2, and v ranges from integers of -N / 2 to N / 2; based on the target fusion function, the inner enface image and the outer enface image are pixel-fused to obtain a full-layer enface image of the fundus image, including: for a pixel at the same position, according to the pixel values ​​of the pixel in the inner enface image and the outer enface image respectively, determining the target piecewise function of the pixel in the target fusion function; substituting the pixel value of the pixel into the target piecewise function to determine the display gray value of the pixel; based on the display gray values ​​of all pixels, determining the full-layer enface image after the inner enface image and the outer enface image are fused.

[0127] On the basis of the above technical solutions, the device further includes: a target training sample determination module, a model parameter correction module, and a model parameter updating module.

[0128] A target training sample determination module obtains a plurality of historical samples to be processed, wherein the historical samples to be processed include historical fundus images, historical inner layer enface images, historical outer layer enface images, and historical full-layer enface images of the same acquisition viewing angle and location; based on the pre-set saturation threshold and the linear mapping function, the historical fundus images, historical inner layer enface images, historical outer layer enface images, and historical full-layer enface images are processed respectively to obtain a training fundus image corresponding to the historical fundus image, a training inner layer enface image corresponding to the historical inner layer enface image, a training outer layer enface image corresponding to the historical outer layer enface image, and a training full-layer enface image corresponding to the historical full-layer enface image; based on the training fundus image, the training inner layer enface image, the training outer layer enface image, and the training full-layer enface image corresponding to each historical sample to be processed, a target training sample is determined;

[0129] A model parameter correction module, for the target training sample, inputs the training fundus image in the target training sample into the processing model to be trained, takes the training inner enface image and the training outer enface image as the output of the processing model to be trained, and corrects the model parameters in the processing model to be trained; and takes the model parameters corresponding to the processing model to be trained when the convergence condition is reached as the model parameters of the target processing model;

[0130] A model parameter updating module inputs a test fundus image in a test sample into the target processing model, and outputs a test inner layer enface image and a test outer layer enface image corresponding to the test fundus image; wherein the test sample includes at least the target training sample; after fusing the test inner layer enface image and the test outer layer enface image based on a target fusion function, a predicted full-layer enface image is obtained; the predicted full-layer enface image and the test full-layer enface image in the test sample are subjected to loss processing, and the model parameters in the target processing model are updated based on the loss value; wherein the loss processing includes determining the gray value loss processing of pixels in the predicted full-layer enface image and the test full-layer enface image.

[0131] The technical solution of the embodiment of the present disclosure obtains a fundus image collected by a fundus camera of an eye part, wherein the fundus image is a grayscale image, and then, based on a preset saturation threshold and a linear mapping function, processes the grayscale value in the fundus image to obtain an image to be used corresponding to the fundus image, further, inputs the image to be used into a pre-trained target processing model, outputs the inner enface image and the outer enface image of the eye part, further, based on a target fusion function, performs pixel fusion on the inner enface image and the outer enface image to obtain a full-layer enface image of the fundus image, and finally, displays the full-layer enface image. Based on a neural network algorithm, the two-dimensional image collected by the fundus camera generates the inner and outer enface images of OCT; and based on the target fusion function, a full-layer enface image is generated. The technical solution of the embodiment of the present disclosure generates an OCT enface image without using OCT, saving user costs and medical resources.

[0132] The image generating device provided in the embodiments of the present disclosure can execute the image generating method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0133] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0134] Embodiment 4

[0135] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 8 , which shows an electronic device (eg, Figure 8 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] like Figure 8As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0137] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0138] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0139] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0140] The electronic device provided in the embodiment of the present disclosure and the image generation method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0141] Embodiment 5

[0142] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image generating method provided by the above embodiment is implemented.

[0143] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0146] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0147] Acquire a fundus image of an eye part collected by a fundus camera, wherein the fundus image is a grayscale image;

[0148] Based on a preset saturation threshold and a linear mapping function, grayscale values ​​in the fundus image are processed to obtain an image to be used corresponding to the fundus image;

[0149] Inputting the image to be used into a pre-trained target processing model, and outputting an inner enface image and an outer enface image of the eye area;

[0150] Perform pixel fusion on the inner enface image and the outer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image;

[0151] The full-thickness enface image is displayed.

[0152] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0153] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0155] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0156] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0157] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0158] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0159] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image generation method, characterized in that: include: Acquire a fundus image of an eye part collected by a fundus camera, wherein the fundus image is a grayscale image; Based on a preset saturation threshold and a linear mapping function, grayscale values ​​in the fundus image are processed to obtain an image to be used corresponding to the fundus image; Inputting the image to be used into a pre-trained target processing model, and outputting an inner enface image and an outer enface image of the eye area; Perform pixel fusion on the inner enface image and the outer enface image based on a target fusion function to obtain a full-layer enface image of the fundus image; The full-thickness enface image is displayed.

2. The method according to claim 1, characterized in that: The grayscale value in the fundus image is processed based on a preset saturation threshold and a linear mapping function to obtain an image to be used corresponding to the fundus image, including: Determining grayscale value statistical information according to the grayscale values ​​of all pixels in the fundus image; Determine a first critical grayscale value and a second critical grayscale value according to a first threshold value and a second threshold value in the saturation threshold value; According to the first critical grayscale value and the second critical grayscale value, the grayscale value of the fundus image is processed to obtain a pixel image to be mapped; The grayscale values ​​in the pixel to be mapped image are processed based on a linear mapping function to obtain the image to be used.

3. The method according to claim 2, characterized in that The step of processing the grayscale value of the fundus image according to the first critical grayscale value and the second critical grayscale value to obtain a pixel image to be mapped includes: For a pixel point in the fundus image, when the grayscale value of the pixel point is less than the first critical grayscale value, adjusting the grayscale value to the first critical grayscale value; When the grayscale value of the pixel point is greater than the second critical grayscale value, the grayscale value is adjusted to the second critical grayscale value.

4. The method according to claim 2, characterized in that: The step of processing the grayscale values ​​in the pixel to be mapped image based on a linear mapping function to obtain the image to be used includes: For a pixel point in the pixel to be mapped image, substituting the grayscale value of the pixel point, the second critical grayscale value and the first critical grayscale value into the linear mapping function to obtain a mapping grayscale value of the pixel point; The image to be used is determined according to the mapping grayscale value of each pixel in the pixel to be mapped image.

5. The method according to claim 1, characterized in that The step of inputting the image to be used into a pre-trained target processing model and outputting an inner enface image and an outer enface image of the eye region comprises: The image to be used is processed based on the main network model and the secondary network model in the target processing model respectively to obtain an inner layer enface image output by the main network model and an outer layer enface image output by the secondary network model.

6. The method according to claim 1, characterized in that The target fusion function is: Among them, I1(x,y) represents the grayscale value of the pixel at the coordinate (x,y) in the inner enface image, I2(x,y) represents the grayscale value of the pixel at the coordinate (x,y) in the outer enface image, w is a weight factor ranging between 0 and 1, M and N are preset positive odd numbers, u and v are temporary variables that take values ​​in rotation, the value range of u is an integer from -M / 2 to M / 2, and the value range of v is an integer from -N / 2 to N / 2.

7. The method according to claim 6, characterized in that The pixel-wise fusion of the inner enface image and the outer enface image based on the target fusion function to obtain a full-layer enface image of the fundus image includes: For a pixel point at the same position, a target piecewise function of the pixel point in the target fusion function is determined according to the pixel values ​​of the pixel point in the inner enface image and the outer enface image respectively; Substituting the pixel value of the pixel point into the target piecewise function to determine the display grayscale value of the pixel point; Based on the displayed grayscale values ​​of all pixels, a full-layer enface image obtained by fusing the inner-layer enface image and the outer-layer enface image is determined.

8. The method according to claim 1, characterized in that Also includes: Acquire multiple historical samples to be processed, wherein the historical samples to be processed include historical fundus images, historical inner enface images, historical outer enface images, and historical full-layer enface images at the same acquisition angle and location; Based on the preset saturation threshold and the linear mapping function, the historical fundus image, the historical inner layer enface image, the historical outer layer enface image and the historical full-layer enface image are processed respectively to obtain a training fundus image corresponding to the historical fundus image, a training inner layer enface image corresponding to the historical inner layer enface image, a training outer layer enface image corresponding to the historical outer layer enface image, and a training full-layer enface image corresponding to the historical full-layer enface image; Based on the training fundus image, the training inner enface image, the training outer enface image and the training full-layer enface image corresponding to each historical sample to be processed, a target training sample is determined.

9. The method according to claim 8, characterized in that After obtaining the target training sample, the method includes: For the target training sample, input the training fundus image in the target training sample into the processing model to be trained, use the training inner enface image and the training outer enface image as outputs of the processing model to be trained, and modify the model parameters in the processing model to be trained; The model parameters corresponding to the processing model to be trained when it reaches the convergence condition are used as the model parameters of the target processing model.

10. The method according to claim 9, characterized in that The method further comprises: Inputting a test fundus image in a test sample into the target processing model, and outputting a test inner layer enface image and a test outer layer enface image corresponding to the test fundus image; wherein the test sample at least includes the target training sample; After fusing the test inner layer enface image and the test outer layer enface image based on the target fusion function, a predicted full layer enface image is obtained; Performing loss processing on the predicted full-layer enface image and the test full-layer enface image in the test sample, and updating model parameters in the target processing model based on the loss value; Among them, the loss processing includes determining the gray value loss processing of pixels in the predicted full-layer enface image and the test full-layer enface image.

Images