Tiny focus recognition method and device based on super-resolution technology and electronic equipment

Abstract

The invention relates to a tiny focus recognition method and device based on a super-resolution technology and electronic equipment. The method comprises the following steps: acquiring at least one first image of a target object; the first image comprises a focus area of the target object; reconstructing each first image through an image reconstruction model to obtain a second image; wherein the image reconstruction model is obtained by updating an observation model through a regular optimization item of the second image; the regularization optimization item is used for indicating regularization of the second image in different dimensions of the same space; the resolution of the first image is lower than that of the second image. By adopting the method, the image details of the second image in different dimensions can be captured and supplemented by regularizing the second image in different dimensions in the same space, so that the second image can clearly display the micro structure and / or boundary information in the focus area of the target object; the definition and detail precision of the second image are improved, and the image reconstruction effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular, to a method and device for identifying tiny lesions based on super-resolution technology. Background Art

[0002] Super-Resolution (SR) reconstruction technology is an image processing technology aimed at recovering the detailed information of high-quality and high-resolution images from observed low-quality and low-resolution images. This technology can significantly improve the image quality and provides important application values for fields such as high-definition television, medical imaging, and remote sensing satellite imaging.

[0003] In the field of medical imaging, how to accurately identify and quantitatively analyze the structure of tiny lesions is particularly crucial for diagnosis and treatment. Traditional super-resolution reconstruction technologies include reconstruction-based and / or deep learning-based methods. Due to the physical limitations of imaging devices and the influence and limitations of different factors during the scanning process, high-frequency information such as edges and textures is ignored, and there are deficiencies in capturing image detail information, making it difficult to clearly display the structure and / or edge information of tiny lesions.

[0004] Therefore, how to accurately model the blur during the image acquisition process to clearly display the structure and / or boundary of tiny lesions is an urgent problem to be solved. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a method, device, and electronic device for identifying tiny lesions based on super-resolution technology.

[0006] In a first aspect, the present application provides a method for identifying tiny lesions based on super-resolution technology, the method comprising: Obtaining at least one first image of a target object; the first image includes a lesion area of the target object; Reconstructing each of the first images through an image reconstruction model to obtain a second image; wherein, the image reconstruction model is obtained by updating an observation model with a regularization optimization term of the second image; the regularization optimization term is used to indicate the regularization of the second image in different dimensions of the same space; the resolution of the first image is lower than the resolution of the second image.

[0007] In one embodiment, the process of establishing the image reconstruction model includes: Establishing the observation model according to downsampling parameters, blurring parameters, and geometric transformation parameters; wherein, the observation model is used to indicate the mapping relationship between the first image and the second image; the expression of the observation model is as follows: ; Among them, is used to indicate the total number of the first images; is used to indicate the first images; is used to indicate the downsampling operator from the high-resolution space to the low-resolution space; is used to indicate the blurring operator; is used to indicate the geometric transformation operator; is used to indicate the second image; Regularize the X-axis dimension, Y-axis dimension, and Z-axis dimension of the second image according to the Manhattan norm to obtain the regularized optimization term; Update the observation model according to the regularized optimization term to obtain the image reconstruction model; among them, the expression of the regularized optimization term is as follows: ; Among them, is used to indicate the regularization function; is used to indicate the balance influence factor; is used to indicate the vector corresponding to the second image in the X-axis dimension; is used to indicate the vector corresponding to the second image in the Y-axis dimension; is used to indicate the vector corresponding to the second image in the Z-axis dimension; The expression of is as shown below: ; Among them, is used to indicate calculating the gradient on .

[0008] In one embodiment, the method for determining the blurring parameter includes: Establish the point spread functions of the first image in the X-axis dimension, Y-axis dimension, and Z-axis dimension respectively by using the Gaussian distribution; Determine the blurring parameter according to the point spread function; among them, the expression of the point spread function is as follows: ; Among them, is used to indicate the coordinate of the first image in the n-axis dimension.

[0009] In one embodiment, the expression of the image reconstruction model is as follows: ; Among them, is used to indicate the first image; is used to indicate the vector corresponding to the first image; is used to indicate the balance influence factor; A vector corresponding to the second image in the X-axis dimension for indication; A vector corresponding to the second image in the Y-axis dimension for indication; A vector corresponding to the second image in the Z-axis dimension for indication.

[0010] In one embodiment, the reconstructing the first image by the image reconstruction model to obtain a second image includes: Determining the mean image of each of the first images as an initial image; Obtaining an initial residual and an initial iteration direction according to the initial image and the expression of the image reconstruction model; Using the conjugate gradient descent algorithm to perform iterative solution on the initial residual in accordance with the initial iteration direction until the residual meets a preset requirement, thereby obtaining the second image.

[0011] In one embodiment, the iterative solution of the expression of the image reconstruction model by using the conjugate gradient descent algorithm and the initial image until the residual meets a preset requirement to obtain the second image includes: In the i-th iteration process, obtaining the iteration direction of the i-th time according to the second alternative image after the (i - 1)-th solution; wherein, i is a positive integer and i > 1; Performing iterative solution according to the second alternative image after the (i - 1)-th solution and the iteration direction of the i-th time to obtain the second alternative image after the i-th solution; Obtaining the i-th residual according to the second alternative image after the i-th solution and the expression of the image reconstruction model; If the i-th residual is less than or equal to a preset threshold, determining the second alternative image after the i-th solution as the second image.

[0012] In one embodiment, the method further includes: Performing three-dimensional reconstruction on the first image and the second image respectively to obtain a first three-dimensional image before super-resolution reconstruction and a second three-dimensional image after super-resolution reconstruction; Subtracting the first three-dimensional image from the second three-dimensional image to obtain a differential three-dimensional image; wherein, the differential three-dimensional image includes at least one differential voxel; Determining the region corresponding to the differential voxel greater than or equal to a preset threshold as a suspicious region and screening out a target region from the suspicious region; Performing three-dimensional reconstruction on the target region again to obtain a target reconstruction region; Overlaying the target reconstruction region onto the second three-dimensional image to obtain a target high-resolution image.

[0013] Second aspect, the present application also provides a tiny lesion recognition device based on super-resolution technology, and the device includes: An acquisition module, configured to acquire at least one first image of a target object; the first image includes a lesion area of the target object; A processing module, configured to reconstruct each of the first images through an image reconstruction model to obtain a second image; Wherein, the image reconstruction model is obtained by updating an observation model with a regularization optimization term of the second image; the regularization optimization term is used to indicate regularization of the second image on different dimensions in the same space; the resolution of the first image is lower than the resolution of the second image.

[0014] Third aspect, the present application also provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the tiny lesion recognition method based on super-resolution technology described in any embodiment of the present application.

[0015] Fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the tiny lesion recognition method based on super-resolution technology described in any embodiment of the present application.

[0016] For the above-mentioned tiny lesion recognition method, device and electronic device based on super-resolution technology, by inputting at least one first image including the lesion area of the target object into the image reconstruction model for reconstruction, a reconstructed second image with high resolution is obtained. In this way, by using the tiny lesion recognition method based on super-resolution technology of the present application, regularization of the second image on different dimensions in the same space can be performed to capture and supplement image details of the second image on different dimensions, so that the second image can clearly display the tiny structure and / or boundary information in the lesion area of the target object, improving the clarity and detail accuracy of the second image and enhancing the image reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an application environment diagram of a tiny lesion recognition method based on super-resolution technology shown according to an exemplary embodiment; Figure 2 is a flowchart of a tiny lesion recognition method based on super-resolution technology shown according to an exemplary embodiment; Figure 3 is a schematic diagram of a first three-dimensional image shown according to an exemplary embodiment; Figure 4 is a schematic diagram of a second three-dimensional image shown according to an exemplary embodiment; Figure 5 It is a schematic diagram of a suspicious area shown according to an exemplary embodiment; Figure 6 It is a schematic diagram of a target reconstruction area shown according to an exemplary embodiment; Figure 7 It is a schematic diagram of a target high - resolution image shown according to an exemplary embodiment; Figure 8A It is a schematic diagram before cross - sectional reconstruction shown according to an exemplary embodiment; Figure 8B It is a schematic diagram after cross - sectional reconstruction shown according to an exemplary embodiment; Figure 9 It is a schematic flowchart of a method for identifying tiny lesions based on super - resolution technology shown according to an exemplary embodiment; Figure 10 It is a structural block diagram of a device for identifying tiny lesions based on super - resolution technology shown according to an exemplary embodiment; Figure 11 It is an internal structure diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0018] In order to make the purpose, technical solutions and advantages of this application more clear and understandable, the following further details this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0019] The terms "first", "second", "third" in the embodiments of this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first", "second", "third" may explicitly or implicitly include at least one of such features. In the description of this application, "at least one" is used to indicate one or more; the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically and clearly defined. In addition, the terms "include" and "have" and any variations thereof are intended to cover non - exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or equipment.

[0020] Reference to "embodiment" in this document means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment each time, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0021] As Figure 1 shown, take the method applied to Figure 1 the electronic device in it as an example for illustration. In the embodiments of this application, the method described is applied to the electronic device and / or the application located in the electronic device; the electronic device can be any mobile terminal or fixed terminal. The terminal can be a device that provides voice and / or data connectivity to the user. Exemplarily, the terminal can be an Internet of Things terminal, such as a sensor device, a mobile phone or a so-called "cellular" phone, and a computer with an Internet of Things terminal. For example, it can be a fixed, portable, pocket-sized, handheld, or computer-integrated device. The application can be, but is not limited to, various software programs, instant apps, and / or mini programs.

[0022] In one embodiment, as Figure 2 shown, a method for identifying minute lesions based on super-resolution technology is provided, including the following steps: S201, obtain at least one first image of the target object; the first image includes the lesion area of the target object.

[0023] In this embodiment, the target object can be any object with vital signs. Optionally, the target object can include, but is not limited to, at least one of humans, animals, and plants.

[0024] In this embodiment, the lesion indicates the tissue or part on the body of the target object where a lesion occurs.

[0025] In some embodiments, when the target object is a human body, the lesion can be located on organs or tissues such as the prostate, kidney, heart, and lungs.

[0026] In this embodiment, the first image can be a low-resolution image in any medical image format.

[0027] Optionally, the medical image format may include, but is not limited to, at least one of the Digital Imaging and Communications in Medicine (DICOM) format, the Neuroimaging Informatics Technology Initiative (NIFTI), the Mayo Medical Imaging (Analyze) format, and the Nearly Raw Raster Data (NRRD) format.

[0028] In some embodiments, acquiring at least one first image of the target object includes at least one of the following: In response to an image acquisition operation, acquiring the first image of the target object through a plurality of acquisition viewing angles; In response to an image acquisition operation, the first image of the target object is acquired through a plurality of acquisition modes.

[0029] Optionally, the acquisition perspective may include but is not limited to at least one of a level perspective, a top perspective, an upward perspective, a side perspective, and an oblique side perspective. For example, the acquisition perspective may be a level perspective and tilted 20° to the right.

[0030] In some embodiments, the electronic device acquires multiple first images of the lesion area containing the target object through acquisition perspectives such as the transverse plane, the coronal plane, the sagittal plane, and the oblique plane.

[0031] In the embodiment of the present application, the acquisition method may include but is not limited to at least one of a spin echo (SE) sequence, a gradient echo (GRE) sequence, a parallel acquisition technique (PAT), a rapid imaging sequence acquisition, and a diffusion weighted imaging technique (DWI).

[0032] S202, reconstructing each of the first images using an image reconstruction model to obtain a second image; The image reconstruction model is obtained by updating the observation model by the regularized optimization term of the second image; the regularized optimization term is used to indicate the regularization of the second image in different dimensions of the same space; and the resolution of the first image is lower than that of the second image.

[0033] In this embodiment, the regularization optimization term is used to indicate that the image variable parameters in the regularization term are extended to different dimensions in the three-dimensional space. The regularization optimization term can simulate the supplementary information of the image details of the second image in different dimensions during the image reconstruction process, so as to achieve denoising and reconstruction effects with different granularities.

[0034] In this embodiment, the observation model can indicate the sampling model during the imaging process; the observation model is used to model the sampling signal to solve the problem of how to better reconstruct the image in the case of incomplete information, noise interference, etc. Through the observation model, the relationship between the sampling signal and the image can be modeled and described, so as to obtain a reconstruction result closer to the real image by mathematical methods. Exemplarily, the observation model can indicate the sampling model during magnetic resonance imaging.

[0035] In this embodiment, the first image is used to indicate the low-resolution image before reconstruction. The second image is used to indicate the high-resolution image after reconstruction.

[0036] In this embodiment, the dimension can be used to indicate different coordinate system establishment directions in the same space. Exemplarily, the dimension can be the X-axis dimension, the Y-axis dimension, and / or the Z-axis dimension, etc.

[0037] In this embodiment, through the regularization optimization term in the image reconstruction model, the supplementary information of the image details of the high-resolution image in different dimensions can be simulated during the image reconstruction process, such as the information of tiny lesion structures. Compared with the traditional method for identifying tiny lesions based on super-resolution technology, the present application can directly reconstruct a high-resolution image that displays the image detail information and supplementary information through the image reconstruction model, improving the accuracy of the image; thus, it can provide more clear and detailed information for clinical diagnosis and treatment.

[0038] In the above method for identifying tiny lesions based on super-resolution technology, by inputting the first image of at least one lesion area containing the target object into the image reconstruction model for reconstruction, a high-resolution second image after reconstruction is obtained. In this way, by using the method for identifying tiny lesions based on super-resolution technology of the present application, through the regularization of the second image in different dimensions in the same space, the capture and supplementation of the image details of the second image in different dimensions can be realized, so that the second image can clearly display the tiny structures and / or boundary information in the lesion area of the target object, improving the clarity and detail accuracy of the second image, and enhancing the image reconstruction effect.

[0039] In some embodiments, the establishment process of the image reconstruction model includes: Establish the observation model according to the downsampling parameter, the blurring parameter, and the geometric transformation parameter; wherein, the observation model is used to indicate the mapping relationship between the first image and the second image; the expression of the observation model is as follows: ; wherein, is used to indicate the total number of the first images; is used to indicate the first image; is used to indicate the downsampling operator from the high-resolution space to the low-resolution space; is used to indicate the blurring operator; is used to indicate the geometric transformation operator; is used to indicate the second image; Regularize the X-axis dimension, Y-axis dimension, and Z-axis dimension of the second image according to the Manhattan norm to obtain the regularized optimization term; Update the observation model according to the regularized optimization term to obtain the image reconstruction model; wherein, the expression of the regularized optimization term is as follows: ; wherein, is used to indicate the regularization function; is used to indicate the balance influence factor; is used to indicate the vector corresponding to the second image in the X-axis dimension; is used to indicate the vector corresponding to the second image in the Y-axis dimension; is used to indicate the vector corresponding to the second image in the Z-axis dimension; The expression of is as follows: ; wherein, is used to indicate calculating the gradient on ;

[0040] In this embodiment, the downsampling operator, also known as the decimation operator, is mainly used to reduce the size of the image and the number of sampling points of the matrix. The process of downsampling usually involves the screening and aggregation of the original image data. The downsampling method may include taking values at every other position and merging regions, etc. Optionally, in the method of taking values at every other position, one point is taken every certain number of points in each row and each column, thereby reducing the amount of image data. Optionally, in the method of merging regions, the pixel values within a certain region are averaged or aggregated by taking the maximum value, etc., to obtain a pixel value representing the region.

[0041] In this embodiment, the blurring operator is mainly used to process the blurriness of the image and achieve the blurring effect of the image. This blurring can be global or local. The blurring operator is used to simulate and reverse the blurring effect in the image.

[0042] In this embodiment, the geometric transformation operator is mainly used to change geometric attributes of an image, such as shape, position, direction, etc. Such a transformation can be global or local, depending on the specific form of the transformation function. For example, affine transformation, projection transformation, etc. can all be regarded as geometric transformation operators. These operators can map pixels in the original image to new positions according to a specific transformation matrix or function, thereby generating a transformed image. By adjusting the transformation parameters, various effects such as image scaling, rotation, translation, and skew can be achieved.

[0043] In this embodiment, the mapping relationship is used to indicate the process of obtaining the first image after the simulated second image undergoes operations of shifting, blurring, and downsampling.

[0044] In some embodiments, establishing the observation model according to the downsampling parameter, the blur parameter, and the geometric transformation parameter includes: Eliminating random noise values in the initial model through the expectation maximization model to obtain the observation model; wherein, the expression of the initial model is as follows: ; Wherein, is used to indicate the total number of the first images; is used to indicate the first image; is used to indicate the downsampling operator from the high-resolution space to the low-resolution space; is used to indicate the blur operator; is used to indicate the geometric transformation operator; is used to indicate the second image; indicates the random noise value.

[0045] In this embodiment, the Manhattan norm (L1 norm) is a norm that measures the length of a vector in the Manhattan space. The Manhattan norm can help the algorithm better handle noise and outliers in the data, thereby improving the robustness and accuracy of the model; and, when dealing with sparse data, the Manhattan norm can usually better preserve the sparsity of the data.

[0046] Exemplarily, in the regularized optimization term can be 0.5.

[0047] In this embodiment, is used to balance the influence between regularization and data fidelity. Exemplarily, = 0.01.

[0048] In some embodiments, the expression of the image reconstruction model is as follows: ; Wherein, For indicating the first image; For indicating the vector corresponding to the first image; For indicating the balance influence factor; For indicating the vector corresponding to the second image in the X-axis dimension; For indicating the vector corresponding to the second image in the Y-axis dimension; For indicating the vector corresponding to the second image in the Z-axis dimension.

[0049] In the embodiments of the present application, by regularizing different dimensions of the second image, supplementary information of the image details of the second image in different dimensions can be simulated. Thus, when reconstructing the second image using the image reconstruction model updated with the regularized optimization term, the tiny structures and / or boundary information of the second image can be clearly displayed, the clarity and detail accuracy of the second image are improved, and the image reconstruction effect is enhanced.

[0050] In some embodiments, the method for determining the blur parameter includes: Establishing point spread functions of the first image in the X-axis dimension, Y-axis dimension, and Z-axis dimension respectively using Gaussian distribution; Determining the blur parameter according to the point spread function; wherein, the expression of the point spread function is as follows: ; Wherein, For indicating the coordinate of the first image in the n-axis dimension.

[0051] In this embodiment, the Gaussian distribution, also known as the Normal distribution, is an important probability distribution in statistics. The normal distribution has the characteristics of a bell-shaped curve, being low at both ends, high in the middle, and symmetric on both sides. Establishing the point spread function based on the Gaussian distribution, on the one hand, the Gaussian distribution has a smooth property, making the point spread process smoother, reducing mutations or jumps, and reducing jagged edges or noise in the image; on the other hand, the Gaussian distribution has differentiability and integrability, which is easy for mathematical modeling and calculation.

[0052] In this embodiment, the point spread function (PSF) is a mathematical function describing the imaging quality of an optical system. It represents the brightness distribution of a point light source on the imaging plane after passing through the optical system. The point spread function describes the response of the imaging system to a point source or point object.

[0053] In this embodiment, the parameter has different sizes in different dimensions. The parameter It can be determined based on multiple experiments. Exemplarily, in the X-axis dimension or the Y-axis dimension, it can be set ; in the Z-axis dimension, it can be set ; where is used to indicate the physical distance between two adjacent first images in the Z-axis dimension.

[0054] In the field of medical imaging, for example, during Magnetic Resonance Imaging (MRI), accurately estimating the PSF is crucial for the quality of image reconstruction because the PSF directly affects the clarity and accuracy of the final synthesized image. Therefore, how to accurately model the blurring in image imaging through the PSF is particularly critical.

[0055] In the embodiments of the present application, when the image is blurred during the imaging process due to certain reasons, since whether the parameters are reasonable directly affects the reconstruction effect of the image details, a Gaussian distribution can be separately established in different dimensions of the same space, and the corresponding parameter sizes of the dimensions can be separately determined, so as to accurately simulate the blurring effect in image imaging, so that the details of the lesion area in the reconstructed second image are clear and specific, and the reconstruction effect of the image details is improved.

[0056] In some embodiments, before reconstructing the first image through the image reconstruction model, the method further includes: Based on each of the first images in the same dimension, determining at least one image stack; where the image stack includes a target image stack; Adjusting the image brightness of each image stack to the target brightness of the target image stack; and / or, adjusting the coordinate system of each image stack to the coordinate system of the target image stack.

[0057] In the embodiments of the present application, by preprocessing each first image (for example, adjusting the image brightness and / or adjusting the coordinate system), the interference caused by inconsistent image brightness or inconsistent image coordinate systems between each first image to image reconstruction can be reduced, thereby further improving the image reconstruction effect and ensuring the clarity of the second image.

[0058] In some embodiments, the reconstructing the first image through the image reconstruction model to obtain a second image includes: Determining the mean image of each of the first images as the initial image; According to the initial image and the expression of the image reconstruction model, obtaining an initial residual and an initial iteration direction; Using the conjugate gradient descent algorithm to iteratively solve the initial residual in the initial iteration direction until the residual meets the preset requirements, and obtaining the second image.

[0059] In this embodiment, the conjugate gradient descent algorithm is an iterative algorithm used to solve the linear equation system AX = b. Here, A is a large sparse matrix indicating the downsampling and blurring process from a high-resolution image (the second image) to a low-resolution image (the first image); X is used to indicate the reconstructed high-resolution image (represented in vector form); and b is used to indicate the observed low-resolution image (represented in vector form). The core of the conjugate gradient descent algorithm is to keep the iterative direction consistent with the conjugate direction of the previous iterative direction.

[0060] Exemplarily, the electronic device determines the initial image X0 according to the average volume of each first image; a method for determining the initial residual is as follows: ; ; where, is used to indicate the balance influence factor; is used to indicate the vector corresponding to the second image in the X-axis dimension; is used to indicate the vector corresponding to the second image in the Y-axis dimension; is used to indicate the vector corresponding to the second image in the Z-axis dimension.

[0061] A method for determining the initial iterative direction is as follows: .

[0062] In some embodiments, the electronic device first inputs the initial image to obtain the image after the first solution; then inputs the obtained image after the first solution to obtain the image after the second solution. The input for each iteration is the solution image obtained in the previous iteration to obtain the latest solution image. Until the residual meets the preset requirements, it is determined that the currently obtained solution image is the second image, and the iterative reconstruction is completed.

[0063] In some embodiments, the iterative solution of the expression of the image reconstruction model using the conjugate gradient descent algorithm and the initial image until the residual meets the preset requirements to obtain the second image includes: In the i-th iteration process, according to the second alternative image after the (i - 1)-th solution, the iterative direction of the i-th iteration is obtained; where, i is a positive integer; i is greater than 1; Performing iterative solution according to the second alternative image after the (i - 1)-th solution and the iterative direction of the i-th iteration to obtain the second alternative image after the i-th solution; According to the second alternative image after the i-th solution and the expression of the image reconstruction model, the i-th residual is obtained; If the i-th residual is less than or equal to a preset threshold, determine the second alternative image after the i-th solution as the second image.

[0064] Exemplarily, a method for determining the i-th iteration direction is as follows: ; wherein, is used to indicate the second alternative image after the (i - 1)-th solution. A method for determining the second alternative image after the i-th solution is ; A method for determining the i-th residual is .

[0065] In the embodiments of the present application, the second image is obtained by iterative solution through the conjugate gradient descent algorithm. The image after the previous solution can be used as the input for the next time, with a relatively fast convergence speed, which can effectively reduce the number of iterations; and it can effectively utilize sparsity, reduce storage requirements, and has high stability.

[0066] In some embodiments, the method further includes: Performing three-dimensional reconstruction on the first image and the second image respectively to obtain a first three-dimensional image before super-resolution reconstruction and a second three-dimensional image after super-resolution reconstruction; Subtracting the first three-dimensional image from the second three-dimensional image to obtain a differential three-dimensional image; wherein, the differential three-dimensional image includes at least one differential voxel; Determine the region corresponding to the differential voxel greater than or equal to the preset threshold as a suspicious region and screen out a target region from the suspicious region; Perform three-dimensional reconstruction on the target region again to obtain a target reconstruction region; Overlay the target reconstruction region on the second three-dimensional image to obtain a target high-resolution image.

[0067] In the embodiments of the present application, three-dimensional reconstruction is used to indicate the process of converting a two-dimensional medical image into a three-dimensional image.

[0068] In some embodiments, three-dimensional reconstruction may include, but is not limited to, at least one of surface reconstruction, volume rendering, and voxel interpolation.

[0069] In the embodiments of the present application, a voxel is the smallest unit that can be segmented in three-dimensional space. The content of a voxel may include, but is not limited to, at least one of density value, label value, opacity, color, and volumetric flow rate.

[0070] In some embodiments, as Figure 3 and Figure 4 shown, Figure 3Schematic diagram of a first three-dimensional image Figure 4 Schematic diagram of a second three-dimensional image. Since the first image is the original sequence image without super-resolution reconstruction, there will be voxel loss or blurring in parts such as edges and details (for example, the tiny structure of the lesion) in the first three-dimensional image obtained by three-dimensional reconstruction of the first image by the electronic device. The second image is the image obtained after super-resolution reconstruction, and parts such as edges and details in the second three-dimensional image obtained by three-dimensional reconstruction of the second image by the electronic device are relatively clear and rich. Therefore, the electronic device can obtain differential voxels by subtracting the second three-dimensional image from the first three-dimensional image to prominently show parts such as edges and details.

[0071] In one embodiment, if the lesion area of the prostate of the target object is included in the first image, the differential voxels may include the lesion area, the noise area generated by super-resolution interpolation, and the tiny structure area of the prostate that is not a lesion.

[0072] Exemplarily, the preset threshold can be 2mm, 3mm, 4mm, etc.

[0073] Exemplarily, as Figure 5 shown, the suspicious area 501 is the tiny structure in the figure, represented by an irregular shape; there may be multiple suspicious areas, and the target area is screened out from the suspicious areas according to images such as Computed Tomography (CT), Positron Emission Tomography (PET), and multi-parametric magnetic resonance.

[0074] In some embodiments, the electronic device divides the target area in different directions to obtain point cloud data for different planes (horizontal plane, sagittal plane, and coronal plane); the electronic device converts the point cloud data into voxel representation and uses the Poisson equation to reconstruct the surface; by solving the Poisson equation, a smooth surface model can be obtained, making the surface model relatively match the point cloud data, so as to "wrap" the point cloud data to highlight the 3D effect of the lesion. Exemplarily, as Figure 6 shown, Figure 6 is a schematic diagram of the target reconstruction area.

[0075] Exemplarily, the target high-resolution image after superimposing the target reconstruction area on the second three-dimensional image is as Figure 7 shown. [[ID=2�]]

[0076] In some embodiments, the electronic device may perform 3D rendering and modeling on a target high-resolution image through volume rendering technology. Volume rendering technology refers to rendering a three-dimensional image or model based on its volume information, thereby achieving realistic transparency, light refraction, and shadow effects. In this way, the structural morphology and adjacent relationships of the lesion can be intuitively and accurately 3D-rendered and finally visually presented.

[0077] Optionally, volume rendering technology is used to indicate a method of displaying information in a three-dimensional space as an image through computer technology. Volume rendering technology can simulate objects and scenes in the real world and use effects such as light, shadow, and material, enabling users to view and interact with data or models more intuitively. Exemplarily, the volume rendering technology may be a volume rendering technology based on OpenGL (Open Graphics Library).

[0078] Traditional two-dimensional images are difficult to accurately describe the three-dimensional structure, geometric shape of organs, and the spatial relationship with surrounding biological tissues. In contrast, in the embodiments of the present application, through processing such as three-dimensional reconstruction of the image, a three-dimensional target high-resolution image is finally obtained, which can accurately display the three-dimensional structure of organs or tissues and realize the visual display of small lesions in organs or tissues; it provides convenience for subsequent clinical diagnosis and treatment. Moreover, the volume rendering technology based on OpenGL can achieve real-time rendering, improve the image processing efficiency, and has good timeliness.

[0079] In some embodiments, the method further includes: Displaying an image reconstruction interface; the image reconstruction interface includes an image display area; the image display area is used to display the first image, the second image, the first three-dimensional image, the second three-dimensional image, the differential three-dimensional image, and / or the target high-resolution image.

[0080] Exemplarily, as Figure 8A and Figure 8B shown. Figure 8A is the original image of the cross-section, that is, the first image; Figure 8B is the image after super-resolution reconstruction of the cross-section, that is, the second image.

[0081] In the embodiments of the present application, the following provides specific examples in combination with any of the above embodiments: Specific Example 1: Figure 9 An exemplary method for identifying small lesions based on super-resolution technology is shown, where the small lesion is a lesion of the prostate. As Figure 9 shown, the method is executed by an electronic device, and the method includes: 1) Obtain image stack data of the prostate in different directions and input it into the image reconstruction model. The image stack in each direction includes multiple first images; here, the direction can be the dimension in the above embodiments.

[0082] 2) Perform data preprocessing on the input image stack data to obtain an initial volume; here, the initial volume can be understood as the initial image in the above embodiments.

[0083] 3) Iteratively solve the initial volume through the conjugate gradient algorithm until a second image is obtained.

[0084] 4) Perform three-dimensional reconstruction on the first image and the second image respectively to obtain a first three-dimensional image before super-resolution reconstruction and a second three-dimensional image after super-resolution reconstruction.

[0085] 5) Subtract the first three-dimensional image from the second three-dimensional image to obtain a differential three-dimensional image; wherein, the differential three-dimensional image includes at least one differential voxel.

[0086] 6) Determine the area corresponding to the differential voxel greater than or equal to the preset threshold as the suspicious area and screen out the target area from the suspicious area.

[0087] 7) Re-perform three-dimensional reconstruction on the target area to obtain a target reconstruction area.

[0088] 8) Superimpose the target reconstruction area on the second three-dimensional image to obtain a target high-resolution image, and perform 3D rendering modeling on the target high-resolution image through volume rendering technology, and perform intuitive and accurate 3D rendering on the structural morphology and adjacent relationship of the prostate and the lesion, and finally visualize and present it, that is, the 3D volume of the nuclear resonance image.

[0089] In the above method for identifying minute lesions based on super-resolution technology, by inputting at least one first image of a lesion area containing a target object into the image reconstruction model for reconstruction, a second high-resolution image after reconstruction is obtained. In this way, by using the method for identifying minute lesions based on super-resolution technology of the present application, the regularization of the second image in different dimensions in the same space can be realized, so as to capture and supplement the image details of the second image in different dimensions, so that the second image can clearly display the minute structure and / or boundary information in the lesion area of the target object, improve the clarity and detail accuracy of the second image, and improve the image reconstruction effect.

[0090] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0091] Based on the same inventive concept, an embodiment of the present application further provides a device for identifying minute lesions based on super-resolution technology for implementing the above-mentioned method for identifying minute lesions based on super-resolution technology. The implementation solution provided by this device for solving problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the device for identifying minute lesions based on super-resolution technology provided below can refer to the limitations on the method for identifying minute lesions based on super-resolution technology in the above text, and will not be repeated here.

[0092] In some embodiments, as Figure 10 shown, the present application further provides a device for identifying minute lesions based on super-resolution technology, and the device includes: An acquisition module 110, configured to acquire at least one first image of a target object; the first image includes a lesion area of the target object; A processing module 120, configured to reconstruct each of the first images through an image reconstruction model to obtain a second image; Wherein, the image reconstruction model is obtained by updating an observation model with a regularization optimization term of the second image; the regularization optimization term is used to indicate the regularization of the second image on different dimensions in the same space; the resolution of the first image is lower than the resolution of the second image.

[0093] In some embodiments, the process of establishing the image reconstruction model includes: Establish the observation model according to downsampling parameters, blurring parameters, and geometric transformation parameters; wherein, the observation model is used to indicate the mapping relationship between the first image and the second image; the expression of the observation model is as follows: ; Wherein, is used to indicate the total number of the first images; is used to indicate the first image; An operator for indicating downsampling from a high-resolution space to a low-resolution space; An operator for indicating a blurring operator; An operator for indicating a geometric transformation operator; An operator for indicating the second image; Regularize the X-axis dimension, Y-axis dimension, and Z-axis dimension of the second image according to the Manhattan norm to obtain the regularized optimization term; Update the observation model according to the regularized optimization term to obtain the image reconstruction model; wherein, the expression of the regularized optimization term is as follows: ; Wherein, An operator for indicating a regularization function; An operator for indicating a balance influence factor; An operator for indicating the vector corresponding to the second image in the X-axis dimension; An operator for indicating the vector corresponding to the second image in the Y-axis dimension; An operator for indicating the vector corresponding to the second image in the Z-axis dimension; The expression of is as follows: ; Wherein, An operator for indicating calculating the gradient on ;

[0094] In some embodiments, the method for determining the blurring parameter includes: Establish point spread functions of the first image in the X-axis dimension, Y-axis dimension, and Z-axis dimension respectively by using a Gaussian distribution; Determine the blurring parameter according to the point spread function; wherein, the expression of the point spread function is as follows: ; Wherein, An operator for indicating the coordinate of the first image in the n-axis dimension.

[0095] In some embodiments, the expression of the image reconstruction model is as follows: ; Wherein, An operator for indicating the first image; An operator for indicating the vector corresponding to the first image; An operator for indicating a balance influence factor; An operator for indicating the vector corresponding to the second image in the X-axis dimension; An operator for indicating the vector corresponding to the second image in the Y-axis dimension; A vector corresponding to the second image in the Z-axis dimension is indicated.

[0096] In some embodiments, the processing module 120 is configured to perform the following steps: Determine the mean image of each of the first images as the initial image; According to the expression of the initial image and the image reconstruction model, obtain an initial residual and an initial iteration direction; Using the conjugate gradient descent algorithm, iteratively solve the initial residual in accordance with the initial iteration direction until the residual meets a preset requirement, and obtain the second image.

[0097] In some embodiments, the processing module 120 is further configured to perform the following steps: In the i-th iteration process, obtain the iteration direction of the i-th time according to the second alternative image after the (i - 1)-th solution; where i is a positive integer; i is greater than 1; Perform iterative solution according to the second alternative image after the (i - 1)-th solution and the iteration direction of the i-th time, and obtain the second alternative image after the i-th solution; According to the expression of the second alternative image after the i-th solution and the image reconstruction model, obtain the i-th residual; If the i-th residual is less than or equal to a preset threshold, determine the second alternative image after the i-th solution as the second image.

[0098] In some embodiments, the device further includes a reconstruction module; the reconstruction module is configured to perform the following steps: Perform three-dimensional reconstruction on the first image and the second image respectively, to obtain a first three-dimensional image before super-resolution reconstruction and a second three-dimensional image after super-resolution reconstruction; Subtract the first three-dimensional image from the second three-dimensional image to obtain a differential three-dimensional image; where the differential three-dimensional image includes at least one differential voxel; Determine the region corresponding to the differential voxel greater than or equal to the preset threshold as a suspicious region and screen out a target region from the suspicious region; Perform three-dimensional reconstruction on the target region again to obtain a target reconstruction region; Overlay the target reconstruction region on the second three-dimensional image to obtain a target high-resolution image.

[0099] Each module in the above image reconstruction device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the electronic device in hardware form or be independent of it, or can be stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0100] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in Figure 11 . The electronic device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for identifying tiny lesions based on super-resolution technology. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the electronic device, or an external keyboard, touchpad, or mouse, etc.

[0101] Those skilled in the art can understand that Figure 11 the structure shown in

[0102] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0103] In one embodiment, an electronic device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented. When the computer program is executed, the steps in the above method embodiments are implemented.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0106] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0107] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for identifying tiny lesions based on super-resolution technology, characterized in that, The method includes: Obtaining at least one first image of a target object; the first image includes a lesion area of the target object; Performing super-resolution reconstruction on each of the first images through an image reconstruction model to obtain a second image; Wherein, the image reconstruction model is obtained by updating an observation model with a regularization optimization term of the second image; the regularization optimization term is used to indicate regularization of the second image in different dimensions of the same space; the resolution of the first image is lower than the resolution of the second image.

2. The method according to claim 1, characterized in that The process of establishing the image reconstruction model includes: Establishing the observation model according to downsampling parameters, blurring parameters, and geometric transformation parameters; wherein, the observation model is used to indicate the mapping relationship between the first image and the second image; the expression of the observation model is as follows: ； Among them, used to indicate the total number of the first images; used to indicate the first images; used to indicate the downsampling operator from the high-resolution space to the low-resolution space; used to indicate the blurring operator; used to indicate the geometric transformation operator; used to indicate the second images; Regularizing the X-axis dimension, Y-axis dimension, and Z-axis dimension of the second image according to the Manhattan norm to obtain the regularization optimization term; Updating the observation model according to the regularization optimization term to obtain the image reconstruction model; wherein, the expression of the regularization optimization term is as follows: ； Among them, is used to indicate the regularization function; is used to indicate the balance influence factor; is used to indicate the vector corresponding to the second image in the X-axis dimension; is used to indicate the vector corresponding to the second image in the Y-axis dimension; is used to indicate the vector corresponding to the second image in the Z-axis dimension; The expression is as follows: ； Among them, used to indicate the calculation of the gradient on.

3. The method according to claim 2, wherein The method for determining the blurring parameters includes: Establishing point spread functions of the first image on the X-axis dimension, Y-axis dimension, and Z-axis dimension respectively using a Gaussian distribution; Determining the blurring parameters according to the point spread functions; wherein, the expression of the point spread function is as follows: ； Among them, is used to indicate the coordinates of the first image in the n-axis dimension.

4. The method according to any one of claims 1 to 3, characterized in that, The expression of the image reconstruction model is as follows: ； wherein, for indicating the first image; for indicating the vector corresponding to the first image; for indicating the balance influence factor; for indicating the vector corresponding to the second image in the X-axis dimension; for indicating the vector corresponding to the second image in the Y-axis dimension; for indicating the vector corresponding to the second image in the Z-axis dimension.

5. The method according to claim 1, wherein Performing super-resolution reconstruction on the first image through the image reconstruction model to obtain a second image, including: Determining the mean image of each of the first images as the initial image; Obtaining an initial residual and an initial iteration direction according to the initial image and the expression of the image reconstruction model; Using the conjugate gradient descent algorithm to perform iterative solution on the initial residual in accordance with the initial iteration direction until the residual meets a preset requirement to obtain the second image.

6. The method according to claim 5, characterized in that Performing iterative solution on the expression of the image reconstruction model using the conjugate gradient descent algorithm and the initial image until the residual meets a preset requirement to obtain the second image, including: In the i-th iteration process, obtaining the iteration direction of the i-th time according to the second alternative image after the (i - 1)-th solution; wherein, i is a positive integer; i is greater than 1; Performing iterative solution according to the second alternative image after the (i - 1)-th solution and the iteration direction of the i-th time to obtain the second alternative image after the i-th solution; Obtaining the i-th residual according to the second alternative image after the i-th solution and the expression of the image reconstruction model; If the i-th residual is less than or equal to a preset threshold, determining the second alternative image after the i-th solution as the second image.

7. The method according to claim 1, characterized in that The method further includes: Performing three-dimensional reconstruction on the first image and the second image respectively to obtain a first three-dimensional image before super-resolution reconstruction and a second three-dimensional image after super-resolution reconstruction; Subtracting the first three-dimensional image from the second three-dimensional image to obtain a differential three-dimensional image; wherein, the differential three-dimensional image includes at least one differential voxel; Determining the region corresponding to the differential voxel greater than or equal to the preset threshold as a suspicious region and screening out a target region from the suspicious region; Perform three-dimensional reconstruction on the target area again to obtain a target reconstructed area; Overlay the target reconstructed area onto the second three-dimensional image to obtain a target high-resolution image.

8. A tiny lesion recognition device based on super-resolution technology, characterized in that, The device includes: An acquisition module, configured to acquire at least one first image of a target object; the first image includes a lesion area of the target object; A processing module, configured to reconstruct each of the first images through an image reconstruction model to obtain a second image; Wherein, the image reconstruction model is obtained by updating an observation model with a regularization optimization term of the second image; the regularization optimization term is used to indicate regularization of the second image in different dimensions of the same space; the resolution of the first image is lower than that of the second image.

9. An electronic device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images