Medical image rendering method and computer equipment

Abstract

The invention relates to the technical field of image rendering, and provides a medical image rendering method and computer equipment, which can reduce noisy points and zigzag edges when rendering an ith frame of medical image. The method comprises the following steps: acquiring an ith frame of medical image and an (i-1) th frame of medical image; inputting the ith frame of medical image into a first neural network to obtain a first image corresponding to the ith frame of medical image; inputting the (i-1) th frame of medical image into a second neural network to obtain a second image corresponding to the (i-1) th frame of medical image; projecting the second image to the first image to obtain a projected second image; and obtaining the rendered ith frame of medical image based on the first image and the projected second image.

Description

Technical Field

[0001] The present application relates to the field of image rendering technology, and in particular to a medical image rendering method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In the medical imaging data visualization project, rendering of medical images is a basic task. Medical image rendering requires real-time rendering because it can provide doctors and researchers with instant feedback, allowing them to immediately view and analyze large amounts of three-dimensional medical image data, thereby making timely diagnosis and treatment decisions.

[0003] Noise and jagged edges are common problems in real-time rendering. Therefore, how to reduce noise and jagged edges when rendering medical images is a technical problem to be solved. Summary of the invention

[0004] Based on this, it is necessary to provide a medical image rendering method, apparatus, computer equipment, storage medium and computer program product to address the above technical problems.

[0005] The present application provides a medical image rendering method, the method comprising:

[0006] Obtaining the i-th frame of medical image and the i-1-th frame of medical image;

[0007] Inputting the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module;

[0008] Inputting the i-1th frame of medical image into a second neural network to obtain a second image corresponding to the i-1th frame of medical image; the second neural network comprises a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module;

[0009] Projecting the second image onto the first image to obtain a projected second image;

[0010] A rendered i-th frame medical image is obtained based on the first image and the projected second image.

[0011] In one embodiment, the first feature extraction module includes: a first multi-scale codec unit and a first kernel prediction network, the output end of an intermediate layer of the first multi-scale codec unit and the output end of each decoder are connected to the input end of the first kernel prediction network; the output end of a decoder of the first multi-scale codec unit is connected to the input end of the first denoising module and the first upsampling module, the output end of the first kernel prediction network is connected to the input end of the first upsampling module, and the output end of the first denoising module is connected to the input end of the first kernel prediction network and the first upsampling module; the inputting the i-th frame of medical image into the first neural network to obtain a first image corresponding to the i-th frame of medical image includes:

[0012] Inputting the i-th frame of medical image into an encoder, obtaining the i-th frame main feature map and the i-th frame auxiliary feature map output by the decoder corresponding to the encoder;

[0013] Inputting the main feature map of the i-th frame into the first denoising module to obtain the main feature map of the i-th frame after denoising;

[0014] Inputting the feature map related to the main feature of the i-th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i-th frame into the first kernel prediction network to output a first convolution kernel weight;

[0015] The denoised main feature map of the i-th frame, the i-th auxiliary feature map and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th medical image frame.

[0016] In one embodiment, the auxiliary features include: motion vectors and normal vectors, the first upsampling module includes: a first motion vector multi-scale filter, a first single-scale filter, a first composite unit, a first bilinear sampling unit, and a fourth kernel prediction network; the denoised i-th frame main feature map, the i-th frame auxiliary feature map, and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th frame medical image, including:

[0017] Inputting the motion vector feature map of the i-th frame and the first convolution kernel weight into the first motion vector multi-scale filter to obtain a filtered motion vector feature map of the i-th frame;

[0018] Inputting the normal vector feature map of the i-th frame into the first single-scale filter to obtain a filtered normal vector feature map of the i-th frame;

[0019] Inputting the filtered motion vector feature map of the i-th frame and the filtered normal vector feature map of the i-th frame into the first composite unit to obtain a first composite feature map;

[0020] Inputting the denoised main feature map of the i-th frame and the first composite feature map into the first bilinear sampling unit to obtain a first sampling feature map;

[0021] The first sampling feature map is input into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame of the medical image.

[0022] In one embodiment, the second feature extraction module includes: a second multi-scale codec unit and a second core prediction network, the output end of an intermediate layer of the second multi-scale codec unit and the output end of each decoder are connected to the input end of the second core prediction network; the output end of a decoder of the second multi-scale codec unit is connected to the input end of the second denoising module and the second upsampling module, the output end of the second core prediction network is connected to the input end of the second upsampling module, and the output end of the second denoising module is connected to the input end of the second core prediction network and the second upsampling module; the inputting the i-1th frame of medical image into the second neural network to obtain a second image corresponding to the i-1th frame of medical image includes:

[0023] Inputting the i-1th frame of medical image into an encoder, and obtaining the i-1th frame of main feature map and the i-1th frame of auxiliary feature map output by the decoder corresponding to the encoder;

[0024] Inputting the i-1th frame main feature map into the second denoising module to obtain the i-1th frame main feature map after denoising;

[0025] Input the feature map related to the main feature of the i-1th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i-1th frame into the second kernel prediction network to output a second convolution kernel weight;

[0026] The denoised i-1th frame main feature map, the i-1th frame auxiliary feature map and the second convolution kernel weight are input into the second upsampling module to obtain a second image corresponding to the i-1th frame medical image.

[0027] In one embodiment, the auxiliary features include: a motion vector and a normal vector, and the second upsampling module includes: a second motion vector multi-scale filter, a second single-scale filter, a second composite unit, a second bilinear sampling unit, and a fifth kernel prediction network;

[0028] The step of inputting the denoised i-1th frame main feature map, the i-1th frame auxiliary feature map, and the second convolution kernel weight into the second upsampling module to obtain a second image corresponding to the i-1th frame medical image includes:

[0029] Inputting the i-1th frame motion vector feature map and the second convolution kernel weight into the second motion vector multi-scale filter to obtain a filtered i-1th frame motion vector feature map;

[0030] Inputting the normal vector feature map of the i-1th frame into the second single-scale filter to obtain a filtered normal vector feature map of the i-1th frame;

[0031] Inputting the filtered i-1th frame motion vector feature map and the filtered i-1th frame normal vector feature map into the second composite unit to obtain a second composite feature map;

[0032] Inputting the denoised i-1th frame main feature map and the second composite feature map into the second bilinear sampling unit to obtain a second sampling feature map;

[0033] The second sampling feature map is input into the fifth kernel prediction network to obtain a second image corresponding to the i-1th frame of medical image.

[0034] In one embodiment, projecting the second image onto the first image comprises:

[0035] The second image is back-warped to the first image based on a motion vector of the second image.

[0036] The present application provides a medical image rendering device, the device comprising:

[0037] An image acquisition module, used for acquiring the i-th frame of medical image and the i-1-th frame of medical image;

[0038] A network processing module, configured to input the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module;

[0039] The network processing module is further used to input the i-1th frame of medical image into a second neural network to obtain a second image corresponding to the i-1th frame of medical image; the second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module;

[0040] A projection module, used for projecting the second image onto the first image to obtain a projected second image;

[0041] The rendered image acquisition module is used to obtain a rendered i-th frame of medical image based on the first image and the projected second image.

[0042] The technical solution of the present application is as follows: first, the i-th frame of medical image is input into the first neural network. Since the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module. Therefore, the jagged edges and artifacts in the i-th frame of medical image can be reduced while removing the noise of the i-th frame of medical image, thereby obtaining a first image corresponding to the i-th frame of medical image. The i-1-th frame of medical image is input into the second neural network. Since the second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module. Therefore, the jagged edges and artifacts in the i-1-th frame of medical image can be reduced while removing the noise of the i-1-th frame of medical image, thereby obtaining a second image corresponding to the i-1-th frame of medical image. The second image is projected onto the first image to obtain the projected second image, and the rendered i-th frame medical image is obtained based on the first image and the projected second image. Since the information of the second image corresponding to the i-1-th frame medical image is utilized and projected onto the first image, and then the rendered i-th frame medical image is obtained based on the projected second image and the first image, on the one hand, the jagged edges and artifacts in the i-th frame medical image obtained after rendering can be further reduced, and on the other hand, the integrity of the i-th frame medical image obtained after rendering is enhanced, thereby greatly improving the quality of the i-th frame medical image rendered.

[0043] Secondly, since the second image is output by the second upsampling module when being projected onto the first image, that is, the second image is a high-resolution image, when the high-resolution second image is projected onto the first image, the influence of low-pass interpolation in the projection process can be reduced, thereby enhancing the sharpness of the finally obtained rendered i-th frame medical image while further reducing the artifacts of the finally obtained rendered i-th frame medical image.

[0044] In addition, the first image is obtained by processing the i-th frame of medical image through the first neural network, the output of the first feature extraction module in the first neural network is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module. The second image is obtained by processing the i-1-th frame of medical image through the second neural network, the output of the second feature extraction module in the second neural network is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module. Since the i-th frame / i-1-th frame of medical image is processed in sequence through the first / second denoising module and the first / second upsampling module, the first / second denoising module and the first / second upsampling module can share intermediate features, intermediate parameters, etc., avoiding overlapping computing resource overhead (or optimizing resource usage), reducing computing time, and improving computing efficiency, thereby greatly improving rendering speed and shortening the time to obtain the rendered i-th frame of medical image.

[0045] The present application provides a medical image rendering method, the method comprising:

[0046] Obtaining the i-th frame of medical image and the i+1-th frame of medical image;

[0047] Inputting the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module;

[0048] Inputting the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image; the third neural network comprises a third feature extraction module, a third denoising module and a third up-sampling module, the output of the third feature extraction module is the input of the third denoising module and the third up-sampling module, and the output of the third denoising module is the input of the third feature extraction module and the third up-sampling module;

[0049] Projecting the third image onto the first image to obtain a projected third image;

[0050] A rendered i-th frame medical image is obtained based on the first image and the projected third image.

[0051] In one embodiment, the third feature extraction module includes: a third multi-scale codec unit and a third core prediction network, the output end of an intermediate layer of the third multi-scale codec unit and the output end of each decoder are connected to the input end of the third core prediction network; the output end of a decoder of the third multi-scale codec unit is connected to the input end of the third denoising module and the third upsampling module, the output end of the third core prediction network is connected to the input end of the third upsampling module, and the output end of the third denoising module is connected to the input end of the third core prediction network and the third upsampling module; the inputting the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image includes:

[0052] Inputting the i+1th frame of medical image into an encoder, and obtaining the i+1th frame of main feature map and the i+1th frame of auxiliary feature map output by the decoder corresponding to the encoder;

[0053] Inputting the i+1th frame main feature map into the third denoising module to obtain the i+1th frame main feature map after denoising;

[0054] Inputting the feature map related to the main feature of the i+1th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i+1th frame into the third kernel prediction network to output the third convolution kernel weight;

[0055] The denoised i+1th frame main feature map, the i+1th frame auxiliary feature map and the third convolution kernel weight are input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image.

[0056] In one embodiment, the auxiliary features include: motion vectors and normal vectors, the third upsampling module includes: a third motion vector multi-scale filter, a third single-scale filter, a third composite unit, a third bilinear sampling unit, and a sixth kernel prediction network; the denoised i+1th frame main feature map, the i+1th frame auxiliary feature map, and the third convolution kernel weight are input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image, including:

[0057] Inputting the i+1th frame motion vector feature map and the third convolution kernel weight into the third motion vector multi-scale filter to obtain a filtered i+1th frame motion vector feature map;

[0058] Inputting the normal vector feature map of the i+1th frame into the third single-scale filter to obtain a filtered normal vector feature map of the i+1th frame;

[0059] Inputting the filtered motion vector feature map of the i+1th frame and the filtered normal vector feature map of the i+1th frame into the third composite unit to obtain a third composite feature map;

[0060] Inputting the denoised (i+1)th frame main feature map and the third composite feature map into the third bilinear sampling unit to obtain a third sampling feature map;

[0061] The third sampling feature map is input into the sixth kernel prediction network to obtain a third image corresponding to the (i+1)th frame of medical image.

[0062] The present application provides a medical image rendering device, the device comprising:

[0063] An image acquisition module, used for acquiring the i-th frame medical image and the i+1-th frame medical image;

[0064] A network processing module, configured to input the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module;

[0065] The network processing module is further used to input the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image; the third neural network includes a third feature extraction module, a third denoising module and a third up-sampling module, the output of the third feature extraction module is the input of the third denoising module and the third up-sampling module, and the output of the third denoising module is the input of the third feature extraction module and the third up-sampling module;

[0066] A projection module, used for projecting the third image onto the first image to obtain a projected third image;

[0067] The rendered image acquisition module is used to obtain a rendered i-th frame of medical image based on the first image and the projected third image.

[0068] The technical solution of the present application is, first, inputting the i-th frame of medical image into the first neural network. Since the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module. Therefore, the jagged edges and artifacts in the i-th frame of medical image can be reduced while removing the noise of the i-th frame of medical image, thereby obtaining a first image corresponding to the i-th frame of medical image. Inputting the i+1-th frame of medical image into the third neural network. Since the third neural network includes a third feature extraction module, a third denoising module and a third upsampling module, the output of the third feature extraction module is the input of the third denoising module and the third upsampling module, and the output of the third denoising module is the input of the third feature extraction module and the third upsampling module. Therefore, the jagged edges and artifacts in the i+1-th frame of medical image can be reduced while removing the noise of the i+1-th frame of medical image, thereby obtaining a third image corresponding to the i+1-th frame of medical image. The third image is projected onto the first image to obtain the projected third image, and the rendered i-th frame medical image is obtained based on the first image and the projected third image. Since the information of the third image corresponding to the i+1-th frame medical image is utilized and projected onto the first image, and then the rendered i-th frame medical image is obtained based on the projected third image and the first image, on the one hand, the jagged edges and artifacts in the i-th frame medical image obtained after rendering can be further reduced, and on the other hand, the integrity of the i-th frame medical image obtained after rendering is enhanced, thereby greatly improving the quality of the i-th frame medical image rendered.

[0069] Secondly, since the third image is output by the third upsampling module when being projected onto the first image, that is, the third image is a high-resolution image, when the high-resolution third image is projected onto the first image, the influence of low-pass interpolation in the projection process can be reduced, thereby enhancing the sharpness of the finally obtained rendered i-th frame medical image while further reducing the artifacts of the finally obtained rendered i-th frame medical image.

[0070] In addition, the first image is obtained by processing the i-th frame of medical image through the first neural network, the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module. The third image is obtained by processing the i+1-th frame of medical image through the third neural network, the third neural network includes a third feature extraction module, a third denoising module and a third upsampling module, the output of the third feature extraction module is the input of the third denoising module and the third upsampling module, and the output of the third denoising module is the input of the third feature extraction module and the third upsampling module. Since the i / i+1-th frame of medical image is processed by the first / third denoising module and the first / third upsampling module in sequence, the first / third denoising module and the first / third upsampling module can share intermediate features, intermediate parameters, etc., avoiding overlapping computing resource overhead (or optimizing resource usage), reducing computing time, and improving computing efficiency, thereby greatly improving rendering speed and shortening the time to obtain the rendered i-th frame of medical image.

[0071] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0072] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute the above method.

[0073] The present application provides a computer program product, on which a computer program is stored, and the computer program is executed by a processor to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic diagram of the process of the medical image rendering method of the present invention;

[0076] Figure 2 A schematic diagram of the structure of a first neural network according to an embodiment of the invention;

[0077] Figure 3 It is a schematic diagram of the structure of a first motion vector multi-scale filter according to an embodiment of the invention;

[0078] Figure 4 Another schematic diagram of the structure of the first neural network according to an embodiment of the invention;

[0079] Figure 5 It is another structural schematic diagram of the first neural network of an embodiment of the invention;

[0080] Figure 6 It is a schematic structural diagram of a first transmittance multi-scale filter according to an embodiment of the invention;

[0081] Figure 7 It is another structural schematic diagram of the first neural network of the embodiment of the invention;

[0082] Figure 8 A schematic diagram of the structure of a second neural network according to an embodiment of the invention;

[0083] Fig. 9 Another schematic diagram of the structure of the second neural network of the embodiment of the invention;

[0084] Fig.10 A schematic diagram of a process of obtaining the i-th frame of a rendered medical image according to the present invention;

[0085] Fig.11 is another schematic diagram of the process of the medical image rendering method of the present invention;

[0086] Fig.12 A schematic diagram of the structure of a third neural network according to an embodiment of the invention;

[0087] Fig.13 Another schematic diagram of the structure of the third neural network of the embodiment of the invention;

[0088] Fig.14 It is a structural block diagram of a medical image rendering device according to an embodiment of the invention;

[0089] Fig.15 A diagram showing the internal structure of a computer device according to an embodiment of the invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0091] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0092] The medical image rendering method provided in the present application can be executed by a computer device, which includes but is not limited to various personal computers and servers. The server can be implemented as an independent server or a server cluster composed of multiple servers.

[0093] The medical image rendering method provided by the present invention can use the information carried by the i-1th frame of medical image to make up for the missing information of the i-th frame of medical image, thereby obtaining the rendered i-th frame of medical image. The rendering method can be applied to the scene of real-time rendering of medical images, and can also be applied to the scene of offline rendering of medical images.

[0094] Figure 1 It is a flowchart of the medical image rendering method of the present invention, such as Figure 1 As shown, the medical image rendering method of the present invention comprises:

[0095] Step S101, obtaining the i-th frame medical image and the i-1-th frame medical image.

[0096] Step S102, inputting the i-th frame of medical image into the first neural network to obtain a first image corresponding to the i-th frame of medical image. The first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module.

[0097] Step S103, inputting the i-1th frame of medical image into the second neural network to obtain a second image corresponding to the i-1th frame of medical image. The second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module.

[0098] Step S104: project the second image onto the first image to obtain a projected second image.

[0099] Step S105 , obtaining a rendered i-th frame of medical image based on the first image and the projected second image.

[0100] The medical image rendering method of the technical solution of the present invention is described below in conjunction with specific embodiments.

[0101] Execute step S101. In this embodiment, with the help of medical imaging technology, such as computed tomography (CT) and magnetic resonance imaging (MRI), the i-1th frame medical image and the i-th frame medical image of a certain part (such as the liver) can be obtained.

[0102] Execute step S102 to obtain a first image corresponding to the i-th frame of the medical image through a first neural network.

[0103] In this embodiment, the first feature extraction module may include: a first multi-scale encoding and decoding unit and a first kernel prediction network.

[0104] Generally speaking, the first multi-scale encoding and decoding unit may include encoders of multiple scales, decoders corresponding thereto, and intermediate layers. Figure 2 , a schematic diagram of the structure of the first neural network in an embodiment of the present invention, in this embodiment, the first multi-scale codec unit may include encoders of four scales (first scale, second scale, third scale and fourth scale) and corresponding decoders (same scale) and two intermediate layers. Specifically, the scales of the first scale codec, the second scale codec, the third scale codec and the fourth scale codec are decreased in sequence.

[0105] Continue to see Figure 2 , the first scale encoder, the second scale encoder, the third scale encoder, the fourth scale encoder, the first intermediate layer, the second intermediate layer, the fourth scale decoder, the third scale decoder, the second scale decoder, and the first scale decoder are connected in sequence; and except for the smallest scale encoder / decoder (the fourth scale encoder and the fourth scale decoder), there are jump connections between encoders and decoders of the same scale, such as Figure 2 As shown, there is a jump connection between the first scale encoder and the first scale decoder, there is a jump connection between the second scale encoder and the second scale decoder, there is a jump connection between the third scale encoder and the third scale decoder, and there is no jump connection between the fourth scale encoder and the fourth scale decoder. This connection method is used between multi-scale encoders / decoders. On the one hand, high-frequency details of the i-th frame of medical image can be restored, and on the other hand, noise interference can be reduced.

[0106] In this embodiment, the output end of the second intermediate layer of the first multi-scale codec unit and the output end of the fourth multi-scale to first multi-scale decoder are connected to the input end of the first core prediction network; the output end of the first multi-scale decoder of the first multi-scale codec unit is connected to the input end of the first denoising module and the first upsampling module, the output end of the first core prediction network is connected to the input end of the first upsampling module, and the output end of the first denoising module is connected to the input end of the first core prediction network and the first upsampling module.

[0107] In this embodiment, the i-th frame of medical image is input to the first encoder, the first decoder outputs the i-th frame main feature map and the i-th frame auxiliary feature map, and the second intermediate layer, the fourth scale decoder, the third scale decoder, the second scale decoder, and the first scale decoder each output a feature map related to the main feature of the i-th frame (the main feature of the i-th frame of medical image). The i-th frame main feature map output by the first decoder is input to the first denoising module to obtain the denoised i-th frame main feature map.

[0108] The feature maps related to the main features of the i-th frame (the main features of the i-th frame of the medical image) output by the second intermediate layer, the fourth scale decoder, the third scale decoder, the second scale decoder, and the first scale decoder and the denoised main feature map of the i-th frame output by the first denoising module are input into the first kernel prediction network and then the first convolution kernel weight is output.

[0109] In this embodiment, when the first feature extraction module includes the above-mentioned first multi-scale codec unit and the first core prediction network, the output end of the intermediate layer of the first multi-scale codec unit and the output end of each decoder are connected to the input end of the first core prediction network; the output end of the decoder of the first multi-scale codec unit is connected to the input end of the first denoising module and the first upsampling module, the output end of the first core prediction network is connected to the input end of the first upsampling module, and the output end of the first denoising module is connected to the input end of the first core prediction network and the first upsampling module.

[0110] That is, the output end of the intermediate layer of the first multi-scale codec unit is connected to the input end of the first core prediction network. The output end of each decoder of the first multi-scale codec unit is connected to the input end of the first core prediction network. The output end of the decoder of the first multi-scale codec unit is connected to the input end of the first denoising module. The output end of the decoder of the first multi-scale codec unit is connected to the input end of the first upsampling module. The output end of the first core prediction network is connected to the input end of the first upsampling module. The output end of the first denoising module is connected to the input end of the first core prediction network. The output end of the first denoising module is connected to the input end of the first upsampling module.

[0111] Inputting the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image, including: inputting the i-th frame of medical image into an encoder to obtain the i-th frame main feature map and the i-th frame auxiliary feature map output by a decoder corresponding to the encoder; inputting the i-th frame main feature map into a first denoising module to obtain the denoised i-th frame main feature map; inputting a feature map related to the i-th frame main feature output by an intermediate layer and each decoder and the denoised i-th frame main feature map into a first kernel prediction network to output a first convolution kernel weight; inputting the denoised i-th frame main feature map, the i-th frame auxiliary feature map and the first convolution kernel weight into a first upsampling module to obtain the first image corresponding to the i-th frame of medical image.

[0112] by Figure 2 The structure of the first neural network shown is introduced as an example. The i-th frame medical image can be input into the first scale encoder to obtain the i-th frame main feature map and the i-th frame auxiliary feature map output by the first scale decoder corresponding to the first scale encoder; the i-th frame main feature map is input into the first denoising module to obtain the denoised i-th frame main feature map; the feature map related to the i-th frame main feature output by the second intermediate layer, the fourth to the first scale decoder and the denoised i-th frame main feature map are input into the first kernel prediction network to output the first convolution kernel weight; the denoised i-th frame main feature map, the i-th frame auxiliary feature map and the first convolution kernel weight are input into the first upsampling module to obtain the first image corresponding to the i-th frame medical image.

[0113] In this embodiment, specifically, after obtaining the i-th frame of medical image, the i-th frame of medical image can be input into the first scale encoder. The i-th frame main feature map and the i-th frame auxiliary feature map output by the first scale encoder can be input into the second scale encoder. As mentioned above, there is a jump connection between the first scale encoder and the first scale decoder, and the i-th frame main feature map and the i-th frame auxiliary feature map output by the first scale encoder can also be input into the first scale decoder.

[0114] The main feature map of the i-th frame and the auxiliary feature map of the i-th frame output by the second scale encoder can be input to the third scale encoder. As mentioned above, there is a jump connection between the second scale encoder and the second scale decoder, and the main feature map of the i-th frame and the auxiliary feature map of the i-th frame output by the second scale encoder can also be input to the second scale decoder.

[0115] The i-th frame main feature map and the i-th frame auxiliary feature map output by the third scale encoder can be input to the fourth scale encoder. As mentioned above, there is a jump connection between the third scale encoder and the third scale decoder, and the i-th frame main feature map and the i-th frame auxiliary feature map output by the third scale encoder can also be input to the third scale decoder.

[0116] The i-th frame main feature map and the i-th frame auxiliary feature map output by the fourth scale encoder may be input to the first intermediate layer. As mentioned above, there is no jump connection between the fourth scale encoder and the fourth scale decoder, so the i-th frame main feature map and the i-th frame auxiliary feature map output by the fourth scale encoder may not be input to the fourth scale decoder.

[0117] The i-th frame main feature map and the i-th frame auxiliary feature map output by the first intermediate layer can be input to the second intermediate layer. The i-th frame main feature map and the i-th frame auxiliary feature map output by the second intermediate layer can be input to the fourth scale decoder. The i-th frame main feature map and the i-th frame auxiliary feature map output by the fourth scale decoder can be input to the third scale decoder. The i-th frame main feature map and the i-th frame auxiliary feature map output by the third scale decoder can be input to the second scale decoder. The i-th frame main feature map and the i-th frame auxiliary feature map output by the second scale decoder can be input to the first scale decoder.

[0118] From the above, it can be known that the first multi-scale encoding and decoding unit includes multiple scale decoders, and these decoders are connected in order from small to large scale. In the main feature map of the i-th frame output by these decoders, the high-frequency details of the main feature map of the i-th frame output by the first scale decoder are well restored. Therefore, the main feature map of the i-th frame output by the first scale decoder can be input to the first denoising module for denoising to obtain the denoised main feature map of the i-th frame.

[0119] In this embodiment, with the help of codecs of different scales of the first multi-scale codec unit, feature coding and decoding processing is performed on the main signal and auxiliary signal of the i-th frame medical image to restore the high-frequency detail information of the image; and the i-th frame main feature map output by the decoder is input into the first denoising module to achieve denoising of the main signal; this embodiment also inputs the denoised i-th frame main feature map output by the first denoising module and the i-th frame main feature map output by each decoder into the first kernel prediction network; the first convolution kernel weight, the denoised i-th frame main feature map and the i-th frame auxiliary feature map are input into the first upsampling module, and the denoised i-th frame main feature map and the i-th frame auxiliary feature map are processed with the help of the first convolution kernel weight, so as to obtain a high-definition first image with less noise.

[0120] In this embodiment, the auxiliary features may include: motion vectors and normal vectors, and the first upsampling module includes: a first motion vector multi-scale filter, a first single-scale filter, a first composite unit, a first bilinear sampling unit, and a fourth kernel prediction network; the denoised main feature map of the i-th frame, the i-th auxiliary feature map, and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th frame medical image, including: inputting the i-th frame motion vector feature map and the first convolution kernel weight into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map; inputting the i-th frame normal vector feature map into the first single-scale filter to obtain the filtered i-th frame normal vector feature map; inputting the filtered i-th frame motion vector feature map and the filtered i-th frame normal vector feature map into the first composite unit to obtain a first composite feature map; inputting the denoised i-th frame main feature map and the first composite feature map into the first bilinear sampling unit to obtain a first sampling feature map; inputting the first sampling feature map into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0121] When the auxiliary features include motion vectors, the first upsampling module includes a motion vector multi-scale filter. For distinction, the motion vector multi-scale filter included in the first upsampling module can be called a first motion vector multi-scale filter; the first motion vector multi-scale filter is used to perform multi-scale filtering on the motion vector feature map.

[0122] Reference Figure 3 , Figure 3 In the example shown, the first motion vector multi-scale filter includes three scale up-sampling modules and three scale down-sampling modules, which are respectively called the first scale down-sampling module, the first scale up-sampling module of the same scale as the first scale down-sampling module, the second scale down-sampling module, the second scale up-sampling module of the same scale as the second scale down-sampling module, the third scale down-sampling module, and the third scale up-sampling module of the same scale as the third scale down-sampling module. There is a jump connection between the down-sampling module and the up-sampling module of the same scale. Figure 3 In the example shown, there is a skip connection between the first scale downsampling module and the first scale upsampling module, there is a skip connection between the second scale downsampling module and the second scale upsampling module, and there is a skip connection between the third scale downsampling module and the third scale upsampling module.

[0123] The first motion vector multi-scale filter further includes: an intermediate layer located between the third scale down-sampling module and the third scale up-sampling module.

[0124] Reference Figure 4After obtaining the first convolution kernel weight output by the first kernel prediction network, the first convolution kernel weight and the i-th frame motion vector feature map output by the first scale decoder can be input into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map.

[0125] The first motion vector multi-scale filter can be used as follows Figure 3 The structure shown, in this case, Figure 3 The input and output between the modules in the first motion vector multi-scale filter are shown as follows:

[0126] The first convolution kernel weight and the i-th frame motion vector feature map output by the first scale decoder are used as the input of the first scale downsampling module, and the output of the first scale downsampling module is used as the input of the second scale downsampling module; since there is a jump connection between the first scale downsampling module and the first scale upsampling module, the output of the first scale downsampling module is also used as the input of the first scale upsampling module; the output of the second scale downsampling module is used as the input of the third scale downsampling module; since there is a jump connection between the second scale downsampling module and the second scale upsampling module, the output of the second scale downsampling module is also used as the input of the second scale upsampling module; the output of the third scale downsampling module is used as the input of the intermediate layer; since there is a jump connection between the third scale downsampling module and the third scale upsampling module, the output of the third scale downsampling module is also used as the input of the third scale upsampling module.

[0127] When the auxiliary feature includes a normal vector, the first upsampling module includes a single-scale filter. For distinction, the single-scale filter included in the first upsampling module can be called a first single-scale filter; the first single-scale filter is used to perform single-scale filtering processing on the normal vector feature map.

[0128] Reference Figure 4 , the normal vector feature map of the i-th frame output by the first scale decoder can be input into the first single-scale filter to obtain the filtered normal vector feature map of the i-th frame.

[0129] Next, the filtered motion vector feature map of the i-th frame and the filtered normal vector feature map of the i-th frame can be input into the first composite unit to obtain a first composite feature map. After obtaining the first composite feature map, the first composite feature map and the denoised main feature map of the i-th frame output by the first denoising module can be input into the first bilinear sampling unit to obtain a first sampling feature map; after obtaining the first sampling feature map, the first sampling feature map can be input into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0130] This embodiment introduces the motion vector in the auxiliary feature, so that motion compensation can be performed during upsampling, and the above embodiment also introduces the normal vector in the auxiliary feature, which can make the first image have a better lighting effect; in addition, in the above embodiment, the motion vector feature map of the i-th frame and the first convolution kernel weight are input into the first motion vector multi-scale filter, and with the help of the first convolution kernel weight and the multi-scale filter, a filtered motion vector feature map of the i-th frame with better filtering effect can be obtained; the normal vector feature map of the i-th frame is input into the first single-scale filter, and while filtering irrelevant information, multi-scale filtering is not required, and a certain processing efficiency is taken into account; the filtered motion vector feature map of the i-th frame and the filtered normal vector feature map of the i-th frame are input into the first composite unit to obtain a first composite feature map, and the denoised i-th frame main feature map and the first composite feature map are input into the first bilinear sampling unit, and the denoised i-th frame main feature map is bilinearly sampled with the help of the first composite feature map to obtain a first sampling feature map, and the first sampling feature map is input into the fourth core prediction network, so that a high-definition and less-noise first image can be obtained.

[0131] In this embodiment, the auxiliary features include: motion vectors, normal vectors and textures, and the first upsampling module includes: a first motion vector multi-scale filter, two first single-scale filters, a first composite unit, a first bilinear sampling unit and a fourth kernel prediction network.

[0132] Inputting the denoised main feature map of the i-th frame, the i-th auxiliary feature map and the first convolution kernel weight into the first upsampling module to obtain a first image corresponding to the i-th frame medical image, including: inputting the i-th frame motion vector feature map and the first convolution kernel weight into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map; inputting the i-th frame normal vector feature map and the i-th frame texture feature map into two first single-scale filters respectively to obtain the filtered i-th frame normal vector feature map and the i-th frame texture feature map; inputting the filtered i-th frame motion vector feature map, the filtered i-th frame normal vector feature map and the filtered i-th frame texture feature map into the first composite unit to obtain a first composite feature map; inputting the denoised main feature map of the i-th frame and the first composite feature map into the first bilinear sampling unit to obtain a first sampling feature map; inputting the first sampling feature map into the fourth kernel prediction network to obtain the first image corresponding to the i-th frame medical image.

[0133] Reference Figure 5 After obtaining the first convolution kernel weight output by the first kernel prediction network, the first convolution kernel weight and the i-th frame motion vector feature map output by the first scale decoder can be input into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map. The first motion vector multi-scale filter can be used as follows Figure 3 The structure shown.

[0134] In the case where the auxiliary features include normal vectors and textures, the first upsampling module includes two single-scale filters, and the two first single-scale filters are used to perform single-scale filtering processing on the normal vector feature map and the texture feature map, respectively.

[0135] Reference Figure 5 The normal vector feature map of the i-th frame output by the first scale decoder can be input into one of the first single-scale filters to obtain the filtered normal vector feature map of the i-th frame; the texture feature map of the i-th frame output by the first scale decoder can also be input into another first single-scale filter to obtain the filtered texture feature map of the i-th frame.

[0136] Next, the filtered motion vector feature map of the i-th frame, the filtered normal vector feature map of the i-th frame, and the filtered texture feature map of the i-th frame can be input into the first composite unit to obtain a first composite feature map. After obtaining the first composite feature map, the first composite feature map and the denoised main feature map of the i-th frame output by the first denoising module can be input into the first bilinear sampling unit to obtain a first sampling feature map; after obtaining the first sampling feature map, the first sampling feature map can be input into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0137] This embodiment introduces the motion vector in the auxiliary feature, so that motion compensation can be performed during upsampling, and the above embodiment also introduces the normal vector in the auxiliary feature, which can make the first image have a better lighting effect, and the above embodiment also introduces the texture in the auxiliary feature, which can improve the image resolution and detail richness, making the image clearer and more realistic; in addition, in the above embodiment, the motion vector feature map of the i-th frame and the first convolution kernel weight are input into the first motion vector multi-scale filter, and with the help of the first convolution kernel weight and the multi-scale filter, the filtered motion vector feature map of the i-th frame with better filtering effect can be obtained; the normal vector feature map of the i-th frame and the i-th frame are input into the first motion vector multi-scale filter, and the first convolution kernel weight and the multi-scale filter are input into the first motion vector multi-scale filter, and ... first convolution kernel weight and the multi-scale filter are input into the first motion vector multi-scale filter, and the first convolution kernel weight and the multi-scale filter are input into the first motion vector multi-scale filter, and the first motion vector feature map of the i-th frame after filtering is input into the first motion vector multi-scale filter, and the first convolution kernel weight and the multi-scale filter are input into the first motion vector multi-scale filter, and the first motion vector feature map of the i-th frame after filtering is input into the first motion vector multi-scale filter, and the first convolution kernel weight and the multi-scale filter are input into the first motion vector multi-scale filter, and the first motion vector feature map of the i-th frame after filtering is input into the first motion vector multi-scale filter, and the first motion vector feature map of the i-th frame after filtering is input into the first motion vector multi-scale filter, and the first motion vector feature map of the i-th The frame texture feature maps are respectively input into two first single-scale filters to filter irrelevant information without multi-scale filtering, taking into account a certain processing efficiency; the filtered i-th frame motion vector feature map, the filtered i-th frame normal vector feature map and the filtered i-th frame texture feature map are input into the first composite unit to obtain a first composite feature map, the denoised i-th frame main feature map and the first composite feature map are input into the first bilinear sampling unit, the denoised i-th frame main feature map is bilinearly sampled with the help of the first composite feature map to obtain a first sampling feature map, and the first sampling feature map is input into the fourth core prediction network, so as to obtain a first image with high definition and less noise.

[0138] In this embodiment, the auxiliary features include: motion vector, normal vector, texture and transmittance, and the first upsampling module includes: a first motion vector multi-scale filter, a first transmittance multi-scale filter, two first single-scale filters, a first composite unit, a first bilinear sampling unit and a fourth kernel prediction network.

[0139] The denoised main feature map of the i-th frame, the i-th auxiliary feature map and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th medical image, including: inputting the i-th motion vector feature map and the first convolution kernel weight into the first motion vector multi-scale filter to obtain the filtered motion vector feature map of the i-th frame; inputting the i-th transmittance feature map and the first convolution kernel weight into the first transmittance multi-scale filter to obtain the filtered transmittance feature map of the i-th frame; inputting the i-th normal vector feature map and the i-th texture feature map into the two first single A scale filter is used to obtain a filtered normal vector feature map of the i-th frame and a filtered texture feature map of the i-th frame; the filtered motion vector feature map of the i-th frame, the filtered transmittance feature map of the i-th frame, the filtered normal vector feature map of the i-th frame and the filtered texture feature map of the i-th frame are input into a first composite unit to obtain a first composite feature map; the denoised i-th frame main feature map and the first composite feature map are input into a first bilinear sampling unit to obtain a first sampling feature map; the first sampling feature map is input into a fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0140] When the auxiliary feature includes transmittance, the first upsampling module includes a transmittance multi-scale filter. For distinction, the transmittance multi-scale filter included in the first upsampling module can be called a first transmittance multi-scale filter; the first transmittance multi-scale filter is used to perform multi-scale filtering processing on the transmittance feature map.

[0141] Reference Figure 6 , Figure 6 An exemplary structure of the first transmittance multiscale filter is shown. In this example, the structure of the first transmittance multiscale filter is consistent with the structure of the first motion vector multiscale filter. For an introduction to the structure of the first transmittance multiscale filter, please refer to the aforementioned content on the structure of the first motion vector multiscale filter.

[0142] Reference Figure 7 After obtaining the first convolution kernel weight output by the first kernel prediction network, the first convolution kernel weight and the i-th frame motion vector feature map output by the first scale decoder can be input into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map. The first convolution kernel weight and the transmittance feature map output by the first scale decoder can also be input into the first transmittance multi-scale filter to obtain the filtered i-th frame transmittance feature map.

[0143] In the case where the auxiliary features include normal vectors and textures, the first upsampling module includes two single-scale filters, and the two first single-scale filters are used to perform single-scale filtering processing on the normal vector feature map and the texture feature map, respectively.

[0144] Reference Figure 7 The normal vector feature map of the i-th frame output by the first scale decoder can be input into one of the first single-scale filters to obtain the filtered normal vector feature map of the i-th frame; the texture feature map of the i-th frame output by the first scale decoder can also be input into another first single-scale filter to obtain the filtered texture feature map of the i-th frame.

[0145] Next, the filtered motion vector feature map of the i-th frame, the filtered transmittance feature map of the i-th frame, the filtered normal vector feature map of the i-th frame, and the filtered texture feature map of the i-th frame can be input into the first composite unit to obtain a first composite feature map. After obtaining the first composite feature map, the first composite feature map and the denoised main feature map of the i-th frame output by the first denoising module can be input into the first bilinear sampling unit to obtain a first sampling feature map; after obtaining the first sampling feature map, the first sampling feature map can be input into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0146] The present embodiment introduces the motion vector in the auxiliary feature, so that motion compensation can be performed during upsampling, and the above embodiment also introduces the normal vector in the auxiliary feature, which can make the first image have a better lighting effect, and the above embodiment also introduces the texture in the auxiliary feature, which can improve the image resolution and detail richness, making the image clearer and more realistic, and the above embodiment also introduces the transmittance in the auxiliary feature, which can make the image clearer and more realistic; in addition, in the above embodiment, the motion vector feature map of the i-th frame and the first convolution kernel weight are input into the first motion vector multi-scale filter, and with the help of the first convolution kernel weight and the multi-scale filter, the filtered motion vector feature map of the i-th frame with better filtering effect can be obtained; the transmittance feature map of the i-th frame and the first convolution kernel weight are input into the first transmittance multi-scale filter, and with the help of the first convolution kernel weight And a multi-scale filter can obtain a filtered i-th frame transmittance feature map with better filtering effect; the i-th frame normal vector feature map and the i-th frame texture feature map are respectively input into two first single-scale filters, and irrelevant information is filtered without multi-scale filtering, while taking into account a certain processing efficiency; the filtered i-th frame motion vector feature map, the filtered i-th frame transmittance feature map, the filtered i-th frame normal vector feature map and the filtered i-th frame texture feature map are input into the first composite unit to obtain a first composite feature map, the denoised i-th frame main feature map and the first composite feature map are input into the first bilinear sampling unit, and the denoised i-th frame main feature map is bilinearly sampled with the help of the first composite feature map to obtain a first sampling feature map, and the first sampling feature map is input into the fourth core prediction network, so that a high-definition first image with less noise can be obtained.

[0147] In this embodiment, the second feature extraction module includes: a second multi-scale codec unit and a second core prediction network, the output end of an intermediate layer of the second multi-scale codec unit and the output end of each decoder are connected to the input end of the second core prediction network; the output end of a decoder of the second multi-scale codec unit is connected to the input end of the second denoising module and the second upsampling module, the output end of the second core prediction network is connected to the input end of the second upsampling module, and the output end of the second denoising module is connected to the input end of the second core prediction network and the second upsampling module.

[0148] That is, the output end of the intermediate layer of the second multi-scale codec unit is connected to the input end of the second core prediction network. The output end of each decoder of the second multi-scale codec unit is connected to the input end of the second core prediction network. The output end of the decoder of the second multi-scale codec unit is connected to the input end of the second denoising module. The output end of the decoder of the second multi-scale codec unit is connected to the input end of the second upsampling module. The output end of the second core prediction network is connected to the input end of the second upsampling module. The output end of the second denoising module is connected to the input end of the second core prediction network. The output end of the second denoising module is connected to the input end of the second upsampling module.

[0149] The structure of the second feature extraction module can be consistent with that of the first feature extraction module. In some scenarios, the second feature extraction module can be used Figure 2 The exemplary structure of the first feature extraction module is shown. At this time, the introduction of the second feature extraction module can refer to the above introduction of the first feature extraction module.

[0150] Inputting the i-1th frame of medical image into a second neural network to obtain a second image corresponding to the i-1th frame of medical image, including: inputting the i-1th frame of medical image into an encoder to obtain the i-1th frame main feature map and the i-1th frame auxiliary feature map output by a decoder corresponding to the encoder; inputting the i-1th frame main feature map into a second denoising module to obtain the denoised i-1th frame main feature map; inputting a feature map related to the i-1th frame main feature output by an intermediate layer and each decoder and the denoised i-1th frame main feature map into a second kernel prediction network to output a second convolution kernel weight; inputting the denoised i-1th frame main feature map, the i-1th frame auxiliary feature map and the second convolution kernel weight into a second upsampling module to obtain the second image corresponding to the i-1th frame of medical image.

[0151] by Figure 8 The structure of the second neural network shown is introduced as an example. The i-1th frame medical image can be input into the first scale encoder to obtain the i-1th frame main feature map and the i-1th frame auxiliary feature map output by the first scale decoder corresponding to the first scale encoder; the i-1th frame main feature map is input into the second denoising module to obtain the denoised i-1th frame main feature map; the feature map related to the i-1th frame main feature output by the second intermediate layer, the fourth to the first scale decoder and the denoised i-1th frame main feature map are input into the second kernel prediction network to output the second convolution kernel weight; the denoised i-1th frame main feature map, the i-1th frame auxiliary feature map and the second convolution kernel weight are input into the second upsampling module to obtain the second image corresponding to the i-1th frame medical image.

[0152] In the following description of this embodiment, the encoder and decoder involved belong to the second multi-scale encoding and decoding unit. Figure 8 After obtaining the i-1th frame of medical image, the i-1th frame of medical image can be input into the first scale encoder. The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the first scale encoder can be input into the second scale encoder. There is a jump connection between the first scale encoder and the first scale decoder, and the i-1th frame main feature map and the i-1th frame auxiliary feature map output by the first scale encoder can also be input into the first scale decoder.

[0153] The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the second scale encoder can be input to the third scale encoder. There is a jump connection between the second scale encoder and the second scale decoder, and the i-1th frame main feature map and the i-1th frame auxiliary feature map output by the second scale encoder can also be input to the second scale decoder.

[0154] The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the third scale encoder can be input to the fourth scale encoder. There is a jump connection between the third scale encoder and the third scale decoder, and the i-1th frame main feature map and the i-1th frame auxiliary feature map output by the third scale encoder can also be input to the third scale decoder.

[0155] The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the fourth scale encoder can be input to the first intermediate layer. There is no jump connection between the fourth scale encoder and the fourth scale decoder, so the i-1th frame main feature map and the i-1th frame auxiliary feature map output by the fourth scale encoder may not be input to the fourth scale decoder.

[0156] The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the first intermediate layer can be input to the second intermediate layer. The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the second intermediate layer can be input to the fourth scale decoder. The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the fourth scale decoder can be input to the third scale decoder. The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the third scale decoder can be input to the second scale decoder. The i-1th frame main feature map and the i-1th frame auxiliary feature map output by the second scale decoder can be input to the first scale decoder.

[0157] The second multi-scale codec unit includes multiple scale decoders, which are connected in order from small to large scales, such as Figure 8 In the example shown, the fourth scale decoder is connected to the third scale decoder, the third scale decoder is connected to the second scale decoder, and the second scale decoder is connected to the first scale decoder. Among the i-1th frame main feature map output by these decoders, the i-1th frame main feature map output by the first scale decoder has better high-frequency details restored. Therefore, the i-1th frame main feature map output by the first scale decoder can be input to the first denoising module for denoising.

[0158] The i-1th frame main feature map output by the first scale decoder can be input into the first denoising module to obtain the i-1th frame main feature map after denoising.

[0159] The denoised i-1th frame main feature map, the i-1th frame main feature map output by the second intermediate layer, and the i-1th frame main feature map output by each scale decoder can be input into the first kernel prediction network to obtain the first convolution kernel weight.

[0160] The denoised i-1th frame main feature map, the first convolution kernel weight, and the i-1th frame auxiliary feature map output by the first scale decoder can be input into the first upsampling module to obtain a first image corresponding to the i-1th frame medical image.

[0161] In this embodiment, with the help of codecs of different scales of the second multi-scale codec unit, feature coding and decoding processing is performed on the main signal and auxiliary signal of the i-1th frame medical image to restore the high-frequency detail information of the image; and the i-1th frame main feature map output by the decoder is input into the second denoising module to achieve denoising of the main signal; this embodiment also inputs the denoised i-1th frame main feature map output by the second denoising module and the i-1th frame main feature map output by each decoder into the second kernel prediction network; the second convolution kernel weight, the denoised i-1th frame main feature map and the i-1th frame auxiliary feature map are input into the second upsampling module, and the denoised i-1th frame main feature map and the i-1th frame auxiliary feature map are processed with the help of the second convolution kernel weight, so as to obtain a high-definition second image with less noise.

[0162] In one embodiment, the auxiliary features include: motion vectors and normal vectors, and the second upsampling module includes: a second motion vector multi-scale filter, a second single-scale filter, a second composite unit, a second bilinear sampling unit, and a fifth kernel prediction network.

[0163] The denoised main feature map of the i-1th frame, the i-1th auxiliary feature map and the second convolution kernel weight are input into the second upsampling module to obtain a second image corresponding to the i-1th frame medical image, including: inputting the i-1th frame motion vector feature map and the second convolution kernel weight into the second motion vector multi-scale filter to obtain the filtered i-1th frame motion vector feature map; inputting the i-1th frame normal vector feature map into the second single-scale filter to obtain the filtered i-1th frame normal vector feature map; inputting the filtered i-1th frame motion vector feature map and the filtered i-1th frame normal vector feature map into the second composite unit to obtain a second composite feature map; inputting the denoised i-1th frame main feature map and the second composite feature map into the second bilinear sampling unit to obtain a second sampling feature map; inputting the second sampling feature map into the fifth kernel prediction network to obtain a second image corresponding to the i-1th frame medical image.

[0164] In the following description of this embodiment, the encoder and decoder involved belong to the second multi-scale encoding and decoding unit. In the case where the auxiliary feature includes a motion vector, the second upsampling module includes a motion vector multi-scale filter. For distinction, the motion vector multi-scale filter included in the second upsampling module can be referred to as a second motion vector multi-scale filter; the second motion vector multi-scale filter is used to perform multi-scale filtering on the motion vector feature map.

[0165] The structure of the second motion vector multi-scale filter can be consistent with the structure of the first motion vector multi-scale filter. In some scenarios, the second motion vector multi-scale filter can be used Figure 3 The structure shown.

[0166] Reference Fig. 9 After obtaining the second convolution kernel weight output by the second kernel prediction network, the second convolution kernel weight and the i-1th frame motion vector feature map output by the first scale decoder can be input into the second motion vector multi-scale filter to obtain the filtered i-1th frame motion vector feature map.

[0167] When the auxiliary feature includes a normal vector, the second upsampling module includes a single-scale filter. For distinction, the single-scale filter included in the second upsampling module can be called a second single-scale filter; the second single-scale filter is used to perform single-scale filtering processing on the normal vector feature map.

[0168] Reference Fig. 9 , the normal vector feature map of the i-1th frame output by the first scale decoder can be input into the second single-scale filter to obtain the filtered normal vector feature map of the i-1th frame.

[0169] Next, the filtered motion vector feature map of the i-1th frame and the filtered normal vector feature map of the i-1th frame can be input into the second composite unit to obtain a second composite feature map. After obtaining the second composite feature map, the second composite feature map and the denoised main feature map of the i-1th frame output by the second denoising module can be input into the second bilinear sampling unit to obtain a second sampling feature map; after obtaining the second sampling feature map, the second sampling feature map can be input into the fifth kernel prediction network to obtain a second image corresponding to the i-1th frame medical image.

[0170] This embodiment introduces the motion vector in the auxiliary feature, so that motion compensation can be performed during upsampling, and the above embodiment also introduces the normal vector in the auxiliary feature, which can make the second image have a better lighting effect; in addition, in the above embodiment, the i-1th frame motion vector feature map and the second convolution kernel weight are input into the second motion vector multi-scale filter, and with the help of the second convolution kernel weight and the multi-scale filter, the filtered i-1th frame motion vector feature map with better filtering effect can be obtained; the i-1th frame normal vector feature map is input into the second single-scale filter, and while filtering irrelevant information, multi-scale filtering is not required, and a certain processing efficiency is taken into account; the filtered i-1th frame motion vector feature map and the filtered i-1th frame normal vector feature map are input into the second composite unit to obtain a second composite feature map, the denoised i-1th frame main feature map and the second composite feature map are input into the second bilinear sampling unit, and the denoised i-1th frame main feature map is bilinearly sampled with the help of the second composite feature map to obtain a second sampling feature map, and the second sampling feature map is input into the fifth core prediction network, so that a high-definition and less-noise second image can be obtained.

[0171] In one embodiment, when the auxiliary features used to obtain the first image corresponding to the i-th frame of the medical image include motion vectors, normal vectors and textures, when the second image corresponding to the i-1-th frame of the medical image is obtained, the auxiliary features used also include motion vectors, normal vectors and textures. The internal structure of the second upsampling module and the input-output relationship between the modules in the second neural network can refer to Figure 5 Examples shown.

[0172] In one embodiment, when the auxiliary features used to obtain the first image corresponding to the i-th frame of the medical image include motion vectors, transmittance, normal vectors and textures, when the second image corresponding to the i-1-th frame of the medical image is obtained, the auxiliary features used also include motion vectors, transmittance, normal vectors and textures. The internal structure of the second upsampling module and the input-output relationship between the modules in the second neural network can refer to Figure 7 Examples shown.

[0173] In this embodiment, projecting the second image onto the first image includes: warping the second image backward onto the first image based on a motion vector of the second image.

[0174] After obtaining the second image and the first image, the second image can be warped backward to the first image according to the motion vector of the second image, and bilinear interpolation processing can be performed during the warping process to obtain the projected second image.

[0175] In this embodiment, based on the motion vector of the second image, the second image can be warped backward to the first image more accurately, so that a more accurate rendered i-th frame medical image can be obtained.

[0176] After obtaining a first image corresponding to the i-th frame of the medical image and a second image corresponding to the i-1-th frame of the medical image, the first image and the second image can be input into a temporal reprojection module to project the second image onto the first image to obtain a projected second image output by the temporal reprojection module.

[0177] The first image and the projected second image may be input into a reconstruction quantization module to obtain an i-th frame of a rendered medical image output by the reconstruction quantization module. In order to improve the real-time rendering speed, the quantization layer in the reconstruction quantization module may be activated using int8-bit weights.

[0178] In the aforementioned step S105, a rendered i-th frame of medical image may be obtained based on the first image and the projected second image.

[0179] Specifically, the difference information between the first image and the projected second image can be used to guide how to adjust the first image to reduce noise and jagged edges. After obtaining the first image and the projected second image, the embodiment of the present application can adjust the first image according to the difference information between the first image and the projected second image, and obtain the rendered i-th frame medical image according to the adjusted first image.

[0180] Through steps S101 to S105, rendering can be performed with the help of the information carried by the i-1th frame of medical image and the neural network including the feature extraction module, the denoising module and the upsampling module, which can effectively reduce the noise and jagged edges of the i-th frame of medical image; moreover, the i-1th frame of medical image and the i-th frame of medical image are respectively input into two neural networks, and the i-1th frame of medical image and the i-th frame of medical image can be processed in parallel to improve the rendering speed; in addition, the present application projects the second image onto the first image, and the second image and the first image are obtained after processing by the upsampling module of the neural network. This processing method of the present application belongs to: first upsampling the medical image to a high resolution and then projecting it, which can reduce the influence of low-pass interpolation during the projection process, thereby protecting the high-frequency information in the medical image, enhancing the image sharpness and reducing image artifacts.

[0181] In some embodiments, in real-time rendering or offline rendering, in order to further reduce the noise and jagged edges of the i-th frame of medical image, the rendered i-th frame of medical image can be obtained by using the i-2-th frame of medical image, the i-1-th frame of medical image and the i-th frame of medical image. In this case, the method provided by the present application may include the following steps:

[0182] Acquire the i-th frame of medical image, the i-1-th frame of medical image and the i-2-th frame of medical image; input the i-th frame of medical image into the first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module; input the i-1-th frame of medical image into the second neural network to obtain a second image corresponding to the i-1-th frame of medical image; the second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module The output of the second denoising module is the input of the second feature extraction module and the second upsampling module; the i-2th frame medical image is input into the fourth neural network to obtain a fourth image corresponding to the i-2th frame medical image; the fourth neural network includes a fourth feature extraction module, a fourth denoising module and a fourth upsampling module, the output of the fourth feature extraction module is the input of the fourth denoising module and the fourth sampling module, and the output of the fourth denoising module is the input of the fourth feature extraction module and the fourth upsampling module; the second image is projected onto the first image to obtain a projected second image; the fourth image is projected onto the first image to obtain a projected fourth image; based on the first image, the projected second image and the projected fourth image, a rendered i-th frame medical image is obtained.

[0183] For the sake of distinction, the neural network used to process the i-2th frame of medical image is called the fourth neural network, the feature extraction module included in the fourth neural network is called the fourth feature extraction module, the denoising module included in the fourth neural network is called the fourth denoising module, and the upsampling module included in the fourth neural network is called the fourth upsampling module.

[0184] The output end of the first feature extraction module is connected to the input end of the first denoising module; the output end of the first feature extraction module is connected to the input end of the first upsampling module; the output end of the first denoising module is connected to the input end of the first feature extraction module; the output end of the first denoising module is connected to the input end of the first upsampling module.

[0185] The output end of the second feature extraction module is connected to the input end of the second denoising module; the output end of the second feature extraction module is connected to the input end of the second upsampling module; the output end of the second denoising module is connected to the input end of the second feature extraction module; the output end of the second denoising module is connected to the input end of the second upsampling module.

[0186] The output end of the fourth feature extraction module is connected to the input end of the fourth denoising module; the output end of the fourth feature extraction module is connected to the input end of the fourth sampling module; the output end of the fourth denoising module is connected to the input end of the fourth feature extraction module; the output end of the fourth denoising module is connected to the input end of the fourth upsampling module.

[0187] After the i-2th frame of medical image is input into the fourth neural network, the i-2th frame of medical image is processed with the help of the fourth feature extraction module, the fourth denoising module and the fourth upsampling module in the fourth neural network to obtain an image output after processing by the fourth neural network, which is called a fourth image.

[0188] In this embodiment, the process of obtaining the fourth image corresponding to the i-2th frame of medical image based on the fourth neural network can refer to the process of obtaining the first image corresponding to the i-th frame of medical image based on the first neural network. Accordingly, the structure of the fourth neural network involved in the processing process can refer to the structure of the first neural network.

[0189] Reference Fig.10 After obtaining a first image corresponding to the i-th frame of medical image, a second image corresponding to the i-1-th frame of medical image, and a fourth image corresponding to the i-2-th frame of medical image, the first image, the second image, and the fourth image can be input into a sequential reprojection module to project the second image onto the first image and project the fourth image onto the first image, thereby obtaining a projected second image and a projected fourth image output by the sequential reprojection module.

[0190] The first image, the projected second image and the projected fourth image may be input into a reconstruction quantization module to obtain an i-th frame of rendered medical image output by the reconstruction quantization module.

[0191] The medical image rendering method provided in the present application can obtain the rendered i-th frame of medical image by using the i-th frame of medical image and the i+1-th frame of medical image. At this time, the method can be used in an offline rendering scenario. The method includes Fig.11 The steps shown are:

[0192] Step S1101, obtaining the i-th frame medical image and the (i+1)-th frame medical image.

[0193] With the help of medical imaging technology (such as CT, MRI), the i-th frame medical image and the i+1-th frame medical image of a certain part (such as the liver) can be obtained.

[0194] Step S1102, input the i-th frame of medical image into the first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module.

[0195] In an embodiment of the present application, after the i-th frame of medical image is input into the first neural network, the i-th frame of medical image is processed with the help of the first feature extraction module, the first denoising module and the first upsampling module in the first neural network to obtain an image output after processing by the first neural network, which is called the first image.

[0196] Step S1103, input the i+1th frame of medical image into the third neural network to obtain a third image corresponding to the i+1th frame of medical image; the third neural network includes a third feature extraction module, a third denoising module and a third upsampling module, the output of the third feature extraction module is the input of the third denoising module and the third upsampling module, and the output of the third denoising module is the input of the third feature extraction module and the third upsampling module.

[0197] For the sake of distinction, the neural network used to process the i+1th frame of medical image is called the third neural network, the feature extraction module included in the third neural network is called the third feature extraction module, the denoising module included in the third neural network is called the third denoising module, and the upsampling module included in the third neural network is called the third upsampling module.

[0198] The output end of the third feature extraction module is connected to the input end of the third denoising module; the output end of the third feature extraction module is connected to the input end of the third upsampling module; the output end of the third denoising module is connected to the input end of the third feature extraction module; the output end of the third denoising module is connected to the input end of the third upsampling module.

[0199] In an embodiment of the present application, after the i+1th frame of medical image is input into the third neural network, the i+1th frame of medical image is processed with the help of the third feature extraction module, the third denoising module and the third upsampling module in the third neural network to obtain an image output after processing by the third neural network, which is called the third image.

[0200] Step S1104: project the third image onto the first image to obtain a projected third image.

[0201] After obtaining the first image and the third image, the third image may be projected onto the first image to obtain a projected third image.

[0202] Step S1105 , obtaining a rendered i-th frame of medical image based on the first image and the projected third image.

[0203] The difference information between the first image and the projected third image can be used to guide how to adjust the first image to reduce noise and jagged edges. After obtaining the first image and the projected third image, the embodiment of the present application can adjust the first image according to the difference information between the first image and the projected third image, and obtain the rendered i-th frame medical image according to the adjusted first image.

[0204] In this embodiment, after obtaining the i-th frame medical image and the i+1-th frame medical image, the i-th frame medical image is input into the first neural network to obtain a first image corresponding to the i-th frame medical image, and the i+1-th frame medical image is input into the third neural network to obtain a third image corresponding to the i+1-th frame medical image; the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module; the third neural network includes a third feature extraction module, a third denoising module and a third upsampling module, the output of the third feature extraction module is the input of the third denoising module and the third upsampling module, and the output of the third denoising module is the input of the third feature extraction module and the third upsampling module; the third image is projected onto the first image to obtain a projected third image; based on the first image and the projected third image, a rendered i-th frame medical image is obtained.

[0205] The present application uses the information carried by the i+1th frame of medical image and a neural network including a feature extraction module, a denoising module and an upsampling module for rendering, which can effectively reduce the noise and jagged edges of the i-th frame of medical image; moreover, the i+1th frame of medical image and the i-th frame of medical image are respectively input into two neural networks, and the i+1th frame of medical image and the i-th frame of medical image can be processed in parallel to improve the rendering speed; in addition, the present application projects the third image onto the first image, and the third image and the first image are obtained after processing by the upsampling module of the neural network. This processing method of the present application belongs to: first upsampling the medical image to a high resolution and then projecting it, which can reduce the influence of low-pass interpolation during the projection process, thereby protecting the high-frequency information in the medical image, enhancing the image sharpness and reducing image artifacts.

[0206] In one embodiment, the third feature extraction module includes: a third multi-scale codec unit and a third core prediction network, the output end of an intermediate layer of the third multi-scale codec unit and the output end of each decoder are connected to the input end of the third core prediction network; the output end of a decoder of the third multi-scale codec unit is connected to the input end of a third denoising module and a third upsampling module, the output end of the third core prediction network is connected to the input end of the third upsampling module, and the output end of the third denoising module is connected to the input end of the third core prediction network and the third upsampling module.

[0207] That is, the output end of the intermediate layer of the third multi-scale codec unit is connected to the input end of the third core prediction network; the output end of each decoder of the third multi-scale codec unit is connected to the input end of the third core prediction network; the output end of the decoder of the third multi-scale codec unit is connected to the input end of the third denoising module; the output end of the decoder of the third multi-scale codec unit is connected to the input end of the third upsampling module; the output end of the third core prediction network is connected to the input end of the third upsampling module; the output end of the third denoising module is connected to the input end of the third core prediction network; the output end of the third denoising module is connected to the input end of the third upsampling module.

[0208] Reference Fig.12 , Fig.12 In the example shown, the third multi-scale codec unit includes four scale codecs, which are respectively called a first scale encoder, a first scale decoder of the same scale as the first scale encoder, a second scale encoder, a second scale decoder of the same scale as the second scale encoder, a third scale encoder, a third scale decoder of the same scale as the third scale encoder, a fourth scale encoder, and a fourth scale decoder of the same scale as the fourth scale encoder, wherein the four scales are arranged in descending order of scale: first scale, second scale, third scale, and fourth scale.

[0209] Except for the smallest scale, there are skip connections between encoders and decoders of the same scale. Fig.12 In the example shown, there is a skip connection between the first scale encoder and the first scale decoder, there is a skip connection between the second scale encoder and the second scale decoder, there is a skip connection between the third scale encoder and the third scale decoder, and there is no skip connection between the fourth scale encoder and the fourth scale decoder, thereby reducing noise interference.

[0210] In the third multi-scale encoding and decoding unit, two intermediate layers are connected in series between the fourth scale encoder and the fourth scale decoder. The intermediate layer close to the fourth scale encoder of the two intermediate layers is called the first intermediate layer, and the intermediate layer far from the fourth scale encoder of the two intermediate layers is called the second intermediate layer.

[0211] Fig.12In the example shown, the input of the second intermediate layer in the third multi-scale encoding and decoding unit and the output of each scale decoder are the input of the third kernel prediction network.

[0212] Inputting the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image, including: inputting the i+1th frame of medical image into an encoder to obtain the i+1th frame main feature map and the i+1th frame auxiliary feature map output by a decoder corresponding to the encoder; inputting the i+1th frame main feature map into a third denoising module to obtain the denoised i+1th frame main feature map; inputting a feature map related to the i+1th frame main feature output by an intermediate layer and each decoder and the denoised i+1th frame main feature map into a third kernel prediction network to output a third convolution kernel weight; inputting the denoised i+1th frame main feature map, the i+1th frame auxiliary feature map and the third convolution kernel weight into a third upsampling module to obtain a third image corresponding to the i+1th frame of medical image.

[0213] by Fig.12 The structure of the third neural network shown is used as an example for introduction. The i+1th frame medical image can be input into the first scale encoder to obtain the i+1th frame main feature map and the i+1th frame auxiliary feature map output by the first scale decoder corresponding to the first scale encoder; the i+1th frame main feature map is input into the third denoising module to obtain the denoised i+1th frame main feature map; the feature map related to the i+1th frame main feature output by the second intermediate layer, the fourth to the first scale decoder and the denoised i+1th frame main feature map are input into the third kernel prediction network to output the third convolution kernel weight; the denoised i+1th frame main feature map, the i+1th frame auxiliary feature map and the third convolution kernel weight are input into the third upsampling module to obtain the third image corresponding to the i+1th frame medical image.

[0214] In the following description of this embodiment, the encoder and decoder involved belong to the third multi-scale encoding and decoding unit.

[0215] After obtaining the i+1th frame of medical image, the i+1th frame of medical image can be input into the first scale encoder. The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the first scale encoder can be input into the second scale encoder. There is a jump connection between the first scale encoder and the first scale decoder, and the i+1th frame main feature map and the i+1th frame auxiliary feature map output by the first scale encoder can also be input into the first scale decoder.

[0216] The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the second scale encoder can be input to the third scale encoder. There is a jump connection between the second scale encoder and the second scale decoder, and the i+1th frame main feature map and the i+1th frame auxiliary feature map output by the second scale encoder can also be input to the second scale decoder.

[0217] The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the third scale encoder can be input to the fourth scale encoder. There is a jump connection between the third scale encoder and the third scale decoder, and the i+1th frame main feature map and the i+1th frame auxiliary feature map output by the third scale encoder can also be input to the third scale decoder.

[0218] The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the fourth scale encoder may be input to the first intermediate layer. There is no jump connection between the fourth scale encoder and the fourth scale decoder, so the i+1th frame main feature map and the i+1th frame auxiliary feature map output by the fourth scale encoder may not be input to the fourth scale decoder.

[0219] The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the first intermediate layer can be input to the second intermediate layer. The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the second intermediate layer can be input to the fourth scale decoder. The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the fourth scale decoder can be input to the third scale decoder. The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the third scale decoder can be input to the second scale decoder. The i+1th frame main feature map and the i+1th frame auxiliary feature map output by the second scale decoder can be input to the first scale decoder.

[0220] The third multi-scale codec unit includes multiple scale decoders, which are connected in order from small to large scales, such as Fig.12 In the example shown, the fourth scale decoder is connected to the third scale decoder, the third scale decoder is connected to the second scale decoder, and the second scale decoder is connected to the first scale decoder. Among the i+1th frame main feature map output by these decoders, the i+1th frame main feature map output by the first scale decoder has better high-frequency details restored. Therefore, the i+1th frame main feature map output by the first scale decoder can be input to the third denoising module for denoising.

[0221] The i+1th frame main feature map output by the first scale decoder can be input into the third denoising module to obtain the i+1th frame main feature map after denoising.

[0222] The denoised i+1th frame main feature map, the i+1th frame main feature map output by the second intermediate layer, and the i+1th frame main feature map output by each scale decoder can be input into the third kernel prediction network to obtain the third convolution kernel weight.

[0223] The denoised i+1th frame main feature map, the first convolution kernel weight, and the i+1th frame auxiliary feature map output by the first scale decoder can be input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image.

[0224] In the above embodiment, with the help of codecs of different scales of the third multi-scale codec unit, feature coding and decoding processing is performed on the main signal and auxiliary signal of the i+1th frame medical image to restore the high-frequency detail information of the image; and the i+1th frame main feature map output by the decoder is input into the third denoising module to achieve denoising of the main signal; this embodiment also inputs the denoised i+1th frame main feature map output by the third denoising module and the i+1th frame main feature map output by each decoder into the third kernel prediction network; the third convolution kernel weight, the denoised i+1th frame main feature map and the i+1th frame auxiliary feature map are input into the third upsampling module, and the denoised i+1th frame main feature map and the i+1th frame auxiliary feature map are processed with the help of the third convolution kernel weight, so as to obtain a high-definition and less noise third image.

[0225] In one embodiment, the auxiliary features include: motion vectors and normal vectors, and the third upsampling module includes: a third motion vector multi-scale filter, a third single-scale filter, a third composite unit, a third bilinear sampling unit, and a sixth kernel prediction network; the denoised i+1th frame main feature map, the i+1th frame auxiliary feature map, and the third convolution kernel weight are input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image, including: inputting the i+1th frame motion vector feature map and the third convolution kernel weight into the third motion vector multi-scale filter to obtain a filter The i+1th frame motion vector feature map after denoising is input; the i+1th frame normal vector feature map is input into the third single-scale filter to obtain the filtered i+1th frame normal vector feature map; the i+1th frame motion vector feature map after filtering and the i+1th frame normal vector feature map after filtering are input into the third composite unit to obtain a third composite feature map; the i+1th frame main feature map after denoising and the third composite feature map are input into the third bilinear sampling unit to obtain a third sampling feature map; the third sampling feature map is input into the sixth kernel prediction network to obtain a third image corresponding to the i+1th frame medical image.

[0226] In the following description of this embodiment, the encoder and decoder involved belong to the third multi-scale encoding and decoding unit.

[0227] When the auxiliary features include motion vectors, the third upsampling module includes a motion vector multi-scale filter. For distinction, the motion vector multi-scale filter included in the third upsampling module can be called a third motion vector multi-scale filter; the third motion vector multi-scale filter is used to perform multi-scale filtering on the motion vector feature map.

[0228] The structure of the third motion vector multi-scale filter can be consistent with the structure of the first motion vector multi-scale filter. In some scenarios, the third motion vector multi-scale filter can be used Figure 3 The structure shown.

[0229] Reference Fig.13 After obtaining the third convolution kernel weight output by the third kernel prediction network, the third convolution kernel weight and the i+1th frame motion vector feature map output by the first scale decoder can be input into the third motion vector multi-scale filter to obtain the filtered i+1th frame motion vector feature map.

[0230] When the auxiliary features include normal vectors, the third upsampling module includes a single-scale filter. For distinction, the single-scale filter included in the third upsampling module can be called a third single-scale filter; the third single-scale filter is used to perform single-scale filtering processing on the normal vector feature map.

[0231] Reference Fig.13 , the normal vector feature map of the i+1th frame output by the first scale decoder can be input into the third single-scale filter to obtain the filtered normal vector feature map of the i+1th frame.

[0232] Next, the filtered motion vector feature map of the i+1th frame and the filtered normal vector feature map of the i+1th frame can be input into the third composite unit to obtain a third composite feature map. After obtaining the third composite feature map, the third composite feature map and the denoised main feature map of the i+1th frame output by the third denoising module can be input into the third bilinear sampling unit to obtain a third sampling feature map; after obtaining the third sampling feature map, the third sampling feature map can be input into the sixth kernel prediction network to obtain a third image corresponding to the i+1th frame medical image.

[0233] The above embodiment introduces the motion vector in the auxiliary feature, so that motion compensation can be performed during upsampling, and the above embodiment also introduces the normal vector in the auxiliary feature, which can make the third image have a better lighting effect; in addition, in the above embodiment, the i+1th frame motion vector feature map and the first convolution kernel weight are input into the third motion vector multi-scale filter, and with the help of the third convolution kernel weight and the multi-scale filter, the filtered i+1th frame motion vector feature map with better filtering effect can be obtained; the i+1th frame normal vector feature map is input into the third single-scale filter, and while filtering irrelevant information, multi-scale filtering is not required, and a certain processing efficiency is taken into account; the filtered i+1th frame motion vector feature map and the filtered i+1th frame normal vector feature map are input into the third composite unit to obtain a third composite feature map, the denoised i+1th frame main feature map and the third composite feature map are input into the third bilinear sampling unit, and the denoised i+1th frame main feature map is bilinearly sampled with the help of the third composite feature map to obtain a third sampling feature map, and the third sampling feature map is input into the sixth core prediction network, so that a high-definition and less-noise third image can be obtained.

[0234] In one embodiment, when the auxiliary features used to obtain the first image corresponding to the i-th frame of the medical image include motion vectors, normal vectors and textures, when the auxiliary features used to obtain the third image corresponding to the i+1-th frame of the medical image also include motion vectors, normal vectors and textures, the internal structure of the third upsampling module and the input-output relationship between the modules in the third neural network can refer to Figure 5 Examples shown.

[0235] In one embodiment, when the auxiliary features used to obtain the first image corresponding to the i-th frame of the medical image include motion vectors, transmittance, normal vectors and textures, when the third image corresponding to the i+1-th frame of the medical image is obtained, the auxiliary features used also include motion vectors, transmittance, normal vectors and textures. The internal structure of the third upsampling module and the input-output relationship between the modules in the third neural network can refer to Figure 7 Examples shown.

[0236] In one embodiment, projecting the third image onto the first image includes: warping the third image back to the first image based on a motion vector of the third image.

[0237] After obtaining the third image and the first image, the third image can be warped backward to the first image according to the motion vector of the third image, and bilinear interpolation processing can be performed during the warping process to obtain the projected third image.

[0238] In this embodiment, based on the motion vector of the third image, the third image can be warped backward to the first image more accurately, so that a more accurate rendered i-th frame medical image can be obtained.

[0239] After obtaining a first image corresponding to the i-th frame of the medical image and a third image corresponding to the i-1-th frame of the medical image, the first image and the third image can be input into a temporal reprojection module to project the third image onto the first image, thereby obtaining a projected third image output by the temporal reprojection module.

[0240] The first image and the projected third image may be input into the reconstruction quantization module to obtain the rendered i-th frame medical image output by the reconstruction quantization module. In order to improve the real-time rendering speed, the quantization layer in the reconstruction quantization module may be activated using int8-bit weights.

[0241] In some embodiments, during offline rendering, in order to further reduce the noise and jagged edges of the i-th frame of the medical image, the rendered i-th frame of the medical image may be obtained by using the i-1-th frame of the medical image, the i-th frame of the medical image, and the i+1-th frame of the medical image. In this case, the method provided by the present application may include the following steps:

[0242] The i-th frame medical image, the i-1-th frame medical image and the i+1-th frame medical image are obtained; the i-th frame medical image is input into a first neural network to obtain a first image corresponding to the i-th frame medical image; the first neural network includes a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module; the i-1-th frame medical image is input into a second neural network to obtain a second image corresponding to the i-1-th frame medical image; the second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module The output of the second denoising module is the input of the second feature extraction module and the second upsampling module; the i+1th frame of medical image is input into the third neural network to obtain a third image corresponding to the i+1th frame of medical image; the third neural network includes a third feature extraction module, a third denoising module and a third upsampling module, the output of the third feature extraction module is the input of the third denoising module and the third upsampling module, and the output of the third denoising module is the input of the third feature extraction module and the third upsampling module; the second image is projected onto the first image to obtain a projected second image; the third image is projected onto the first image to obtain a projected third image; based on the first image, the projected second image and the projected third image, a rendered i-th frame of medical image is obtained.

[0243] In this embodiment, the process of obtaining the third image corresponding to the (i+1)th frame of medical image based on the third neural network can refer to the process of obtaining the first image corresponding to the (i)th frame of medical image based on the first neural network. Accordingly, the structure of the third neural network involved in the processing process can refer to the structure of the first neural network.

[0244] After obtaining a first image corresponding to the i-th frame of medical image, a second image corresponding to the i-1-th frame of medical image, and a third image corresponding to the i+1-th frame of medical image, the first image, the second image, and the third image can be input into a temporal reprojection module to project the second image onto the first image and project the third image onto the first image, thereby obtaining a projected second image and a projected third image output by the temporal reprojection module.

[0245] The first image, the projected second image and the projected third image may be input into a reconstruction quantization module to obtain an i-th frame of rendered medical image output by the reconstruction quantization module.

[0246] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0247] Based on the same inventive concept, the embodiment of the present application also provides a medical image rendering device for implementing the medical image rendering method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more medical image rendering device embodiments provided below can refer to the limitations of the medical image rendering method above, and will not be repeated here.

[0248] In one embodiment, the present application provides a medical image rendering device, the device comprising: Fig.14 The modules shown are, specifically:

[0249] An image acquisition module 1401 is used to acquire an i-th medical image frame and an i-1-th medical image frame;

[0250] The network processing module 1402 is used to input the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network includes a first feature extraction module, a first denoising module and a first up-sampling module, the output of the first feature extraction module is the input of the first denoising module and the first up-sampling module, and the output of the first denoising module is the input of the first feature extraction module and the first up-sampling module;

[0251] The network processing module 1402 is further used to input the i-1th frame of medical image into a second neural network to obtain a second image corresponding to the i-1th frame of medical image; the second neural network includes a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module;

[0252] A projection module 1403 is used to project the second image onto the first image to obtain a projected second image;

[0253] The rendered image acquisition module 1404 is used to obtain a rendered i-th frame of medical image based on the first image and the projected second image.

[0254] In one embodiment, the first feature extraction module includes: a first multi-scale codec unit and a first core prediction network, the output end of an intermediate layer of the first multi-scale codec unit and the output end of each decoder are connected to the input end of the first core prediction network; the output end of the first multi-scale codec unit-decoder is connected to the input end of the first denoising module and the first upsampling module, the output end of the first core prediction network is connected to the input end of the first upsampling module, and the output end of the first denoising module is connected to the input end of the first core prediction network and the first upsampling module; the network processing module 1402 is further used to: The i-th frame medical image is input into an encoder to obtain the i-th frame main feature map and the i-th frame auxiliary feature map output by the decoder corresponding to the encoder; the i-th frame main feature map is input into the first denoising module to obtain the denoised i-th frame main feature map; the feature maps related to the i-th frame main features output by the intermediate layer and each decoder and the denoised i-th frame main feature map are input into the first kernel prediction network to output the first convolution kernel weight; the denoised i-th frame main feature map, the i-th frame auxiliary feature map and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th frame medical image.

[0255] In one embodiment, the auxiliary features include: motion vectors and normal vectors, and the first upsampling module includes: a first motion vector multi-scale filter, a first single-scale filter, a first composite unit, a first bilinear sampling unit and a fourth kernel prediction network; the network processing module 1402 is also used to: input the i-th frame motion vector feature map and the first convolution kernel weight into the first motion vector multi-scale filter to obtain the filtered i-th frame motion vector feature map; input the i-th frame normal vector feature map into the first single-scale filter to obtain the filtered i-th frame normal vector feature map; input the filtered i-th frame motion vector feature map and the filtered i-th frame normal vector feature map into the first composite unit to obtain a first composite feature map; input the denoised i-th frame main feature map and the first composite feature map into the first bilinear sampling unit to obtain a first sampling feature map; input the first sampling feature map into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame medical image.

[0256] In one embodiment, the second feature extraction module includes: a second multi-scale codec unit and a second core prediction network, the output end of the second multi-scale codec unit-intermediate layer and the output end of each decoder are connected to the input end of the second core prediction network; the output end of the second multi-scale codec unit-decoder is connected to the input end of the second denoising module and the second upsampling module, the output end of the second core prediction network is connected to the input end of the second upsampling module, and the output end of the second denoising module is connected to the input end of the second core prediction network and the second upsampling module; the network processing module 1402 is also used to: convert the i-1th frame of medical image An encoder is input to obtain the main feature map of the i-1th frame and the auxiliary feature map of the i-1th frame output by the decoder corresponding to the encoder; the main feature map of the i-1th frame is input to the second denoising module to obtain the denoised main feature map of the i-1th frame; the feature maps related to the main features of the i-1th frame output by the intermediate layer and each decoder and the denoised main feature map of the i-1th frame are input to the second kernel prediction network to output the second convolution kernel weight; the denoised main feature map of the i-1th frame, the auxiliary feature map of the i-1th frame and the second convolution kernel weight are input to the second upsampling module to obtain a second image corresponding to the i-1th medical image.

[0257] In one embodiment, the auxiliary features include: motion vectors and normal vectors, and the second upsampling module includes: a second motion vector multi-scale filter, a second single-scale filter, a second composite unit, a second bilinear sampling unit and a fifth kernel prediction network; the network processing module 1402 is also used to: input the i-1th frame motion vector feature map and the second convolution kernel weight into the second motion vector multi-scale filter to obtain the filtered i-1th frame motion vector feature map; input the i-1th frame normal vector feature map into the second single-scale filter to obtain the filtered i-1th frame normal vector feature map; input the filtered i-1th frame motion vector feature map and the filtered i-1th frame normal vector feature map into the second composite unit to obtain a second composite feature map; input the denoised i-1th frame main feature map and the second composite feature map into the second bilinear sampling unit to obtain a second sampling feature map; input the second sampling feature map into the fifth kernel prediction network to obtain a second image corresponding to the i-1th frame medical image.

[0258] In one embodiment, the projection module 1403 is further configured to: warp the second image backward to the first image based on a motion vector of the second image.

[0259] In one embodiment, the present application provides a medical image rendering device, the device comprising: Fig.14 The modules shown are, specifically:

[0260] An image acquisition module 1401 is used to acquire an i-th frame of medical image and an i+1-th frame of medical image;

[0261] The network processing module 1402 is used to input the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network includes a first feature extraction module, a first denoising module and a first up-sampling module, the output of the first feature extraction module is the input of the first denoising module and the first up-sampling module, and the output of the first denoising module is the input of the first feature extraction module and the first up-sampling module;

[0262] The network processing module 1402 is further used to input the (i+1)th frame of medical image into a third neural network to obtain a third image corresponding to the (i+1)th frame of medical image; the third neural network includes a third feature extraction module, a third denoising module and a third up-sampling module, the output of the third feature extraction module is the input of the third denoising module and the third up-sampling module, and the output of the third denoising module is the input of the third feature extraction module and the third up-sampling module;

[0263] A projection module 1403 is used to project the third image onto the first image to obtain a projected third image;

[0264] The rendered image acquisition module 1404 is used to obtain a rendered i-th frame of medical image based on the first image and the projected third image.

[0265] In one embodiment, the third feature extraction module includes: a third multi-scale codec unit and a third core prediction network, the output end of the intermediate layer of the third multi-scale codec unit and the output end of each decoder are connected to the input end of the third core prediction network; the output end of the decoder of the third multi-scale codec unit is connected to the input end of the third denoising module and the third upsampling module, the output end of the third core prediction network is connected to the input end of the third upsampling module, and the output end of the third denoising module is connected to the input end of the third core prediction network and the third upsampling module; the network processing module 1402 is also used to: convert the i+1th frame of medical image An encoder is input to obtain the main feature map of the i+1th frame and the auxiliary feature map of the i+1th frame output by the decoder corresponding to the encoder; the main feature map of the i+1th frame is input to the third denoising module to obtain the denoised main feature map of the i+1th frame; the feature maps related to the main features of the i+1th frame output by the intermediate layer and each decoder and the denoised main feature map of the i+1th frame are input to the third kernel prediction network to output the third convolution kernel weight; the denoised main feature map of the i+1th frame, the auxiliary feature map of the i+1th frame and the third convolution kernel weight are input to the third upsampling module to obtain a third image corresponding to the i+1th medical image.

[0266] In one embodiment, the auxiliary features include: motion vectors and normal vectors, and the third upsampling module includes: a third motion vector multi-scale filter, a third single-scale filter, a third composite unit, a third bilinear sampling unit and a sixth kernel prediction network; the network processing module 1402 is further used to: input the i+1th frame motion vector feature map and the third convolution kernel weight into the third motion vector multi-scale filter to obtain the filtered i+1th frame motion vector feature map; input the i+1th frame normal vector feature map into the third single-scale filter to obtain the filtered i+1th frame normal vector feature map; input the filtered i+1th frame motion vector feature map and the filtered i+1th frame normal vector feature map into the third composite unit to obtain a third composite feature map; input the denoised i+1th frame main feature map and the third composite feature map into the third bilinear sampling unit to obtain a third sampling feature map; input the third sampling feature map into the sixth kernel prediction network to obtain a third image corresponding to the i+1th frame medical image.

[0267] Each module in the above-mentioned medical image rendering device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0268] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Fig.15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a medical image rendering method is implemented.

[0269] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0270] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0271] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0272] In one embodiment, a computer program product is provided, on which a computer program is stored, and the computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0273] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0274] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0275] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0276] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A medical image rendering method, characterized in that: The method comprises: Obtaining the i-th frame of medical image and the i-1-th frame of medical image; Inputting the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module; Inputting the i-1th frame of medical image into a second neural network to obtain a second image corresponding to the i-1th frame of medical image; the second neural network comprises a second feature extraction module, a second denoising module and a second upsampling module, the output of the second feature extraction module is the input of the second denoising module and the second upsampling module, and the output of the second denoising module is the input of the second feature extraction module and the second upsampling module; Projecting the second image onto the first image to obtain a projected second image; A rendered i-th frame medical image is obtained based on the first image and the projected second image.

2. The method according to claim 1, characterized in that: The first feature extraction module includes: a first multi-scale codec unit and a first core prediction network, the output end of an intermediate layer of the first multi-scale codec unit and the output end of each decoder are connected to the input end of the first core prediction network; the output end of a decoder of the first multi-scale codec unit is connected to the input end of the first denoising module and the first upsampling module, the output end of the first core prediction network is connected to the input end of the first upsampling module, and the output end of the first denoising module is connected to the input end of the first core prediction network and the first upsampling module; the inputting the i-th frame of medical image into the first neural network to obtain a first image corresponding to the i-th frame of medical image includes: Inputting the i-th frame of medical image into an encoder, obtaining the i-th frame main feature map and the i-th frame auxiliary feature map output by the decoder corresponding to the encoder; Inputting the main feature map of the i-th frame into the first denoising module to obtain the main feature map of the i-th frame after denoising; Inputting the feature map related to the main feature of the i-th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i-th frame into the first kernel prediction network to output a first convolution kernel weight; The denoised main feature map of the i-th frame, the i-th auxiliary feature map and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th medical image frame.

3. The method according to claim 2, characterized in that The auxiliary features include: motion vectors and normal vectors; the first upsampling module includes: a first motion vector multi-scale filter, a first single-scale filter, a first composite unit, a first bilinear sampling unit, and a fourth kernel prediction network; the denoised i-th frame main feature map, the i-th frame auxiliary feature map, and the first convolution kernel weight are input into the first upsampling module to obtain a first image corresponding to the i-th frame medical image, including: Inputting the motion vector feature map of the i-th frame and the first convolution kernel weight into the first motion vector multi-scale filter to obtain a filtered motion vector feature map of the i-th frame; Inputting the normal vector feature map of the i-th frame into the first single-scale filter to obtain a filtered normal vector feature map of the i-th frame; Inputting the filtered motion vector feature map of the i-th frame and the filtered normal vector feature map of the i-th frame into the first composite unit to obtain a first composite feature map; Inputting the denoised main feature map of the i-th frame and the first composite feature map into the first bilinear sampling unit to obtain a first sampling feature map; The first sampling feature map is input into the fourth kernel prediction network to obtain a first image corresponding to the i-th frame of the medical image.

4. The method according to any one of claims 1 to 3, characterized in that The second feature extraction module includes: a second multi-scale codec unit and a second core prediction network, the output end of an intermediate layer of the second multi-scale codec unit and the output end of each decoder are connected to the input end of the second core prediction network; the output end of a decoder of the second multi-scale codec unit is connected to the input end of the second denoising module and the second upsampling module, the output end of the second core prediction network is connected to the input end of the second upsampling module, and the output end of the second denoising module is connected to the input end of the second core prediction network and the second upsampling module; the inputting the i-1th frame of medical image into the second neural network to obtain a second image corresponding to the i-1th frame of medical image includes: Inputting the i-1th frame of medical image into an encoder, and obtaining the i-1th frame of main feature map and the i-1th frame of auxiliary feature map output by the decoder corresponding to the encoder; Inputting the i-1th frame main feature map into the second denoising module to obtain the i-1th frame main feature map after denoising; Input the feature map related to the main feature of the i-1th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i-1th frame into the second kernel prediction network to output a second convolution kernel weight; The denoised i-1th frame main feature map, the i-1th frame auxiliary feature map and the second convolution kernel weight are input into the second upsampling module to obtain a second image corresponding to the i-1th frame medical image.

5. The method according to claim 4, characterized in that The auxiliary features include: motion vectors and normal vectors, and the second upsampling module includes: a second motion vector multi-scale filter, a second single-scale filter, a second composite unit, a second bilinear sampling unit, and a fifth kernel prediction network; The step of inputting the denoised i-1th frame main feature map, the i-1th frame auxiliary feature map, and the second convolution kernel weight into the second upsampling module to obtain a second image corresponding to the i-1th frame medical image includes: Inputting the i-1th frame motion vector feature map and the second convolution kernel weight into the second motion vector multi-scale filter to obtain a filtered i-1th frame motion vector feature map; Inputting the normal vector feature map of the i-1th frame into the second single-scale filter to obtain a filtered normal vector feature map of the i-1th frame; Inputting the filtered i-1th frame motion vector feature map and the filtered i-1th frame normal vector feature map into the second composite unit to obtain a second composite feature map; Inputting the denoised i-1th frame main feature map and the second composite feature map into the second bilinear sampling unit to obtain a second sampling feature map; The second sampling feature map is input into the fifth kernel prediction network to obtain a second image corresponding to the i-1th frame of medical image.

6. The method according to claim 1, characterized in that Projecting the second image onto the first image comprises: The second image is back-warped to the first image based on a motion vector of the second image.

7. A medical image rendering method, characterized in that: include: Obtaining the i-th frame of medical image and the i+1-th frame of medical image; Inputting the i-th frame of medical image into a first neural network to obtain a first image corresponding to the i-th frame of medical image; the first neural network comprises a first feature extraction module, a first denoising module and a first upsampling module, the output of the first feature extraction module is the input of the first denoising module and the first upsampling module, and the output of the first denoising module is the input of the first feature extraction module and the first upsampling module; Inputting the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image; the third neural network comprises a third feature extraction module, a third denoising module and a third up-sampling module, the output of the third feature extraction module is the input of the third denoising module and the third up-sampling module, and the output of the third denoising module is the input of the third feature extraction module and the third up-sampling module; Projecting the third image onto the first image to obtain a projected third image; A rendered i-th frame medical image is obtained based on the first image and the projected third image.

8. The method according to claim 7, characterized in that The third feature extraction module includes: a third multi-scale codec unit and a third kernel prediction network, the output end of an intermediate layer of the third multi-scale codec unit and the output end of each decoder are connected to the input end of the third kernel prediction network; the output end of a decoder of the third multi-scale codec unit is connected to the input end of the third denoising module and the third up-sampling module, the output end of the third kernel prediction network is connected to the input end of the third up-sampling module, and the output end of the third denoising module is connected to the input end of the third kernel prediction network and the third up-sampling module; the inputting the i+1th frame of medical image into a third neural network to obtain a third image corresponding to the i+1th frame of medical image includes: Inputting the i+1th frame of medical image into an encoder, and obtaining the i+1th frame of main feature map and the i+1th frame of auxiliary feature map output by the decoder corresponding to the encoder; Inputting the i+1th frame main feature map into the third denoising module to obtain the i+1th frame main feature map after denoising; Inputting the feature map related to the main feature of the i+1th frame output by the first intermediate layer and each decoder and the denoised main feature map of the i+1th frame into the third kernel prediction network to output the third convolution kernel weight; The denoised i+1th frame main feature map, the i+1th frame auxiliary feature map and the third convolution kernel weight are input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image.

9. The method according to claim 8, characterized in that The auxiliary features include: motion vectors and normal vectors, the third upsampling module includes: a third motion vector multi-scale filter, a third single-scale filter, a third composite unit, a third bilinear sampling unit and a sixth kernel prediction network; the denoised i+1th frame main feature map, the i+1th frame auxiliary feature map and the third convolution kernel weight are input into the third upsampling module to obtain a third image corresponding to the i+1th frame medical image, including: Inputting the i+1th frame motion vector feature map and the third convolution kernel weight into the third motion vector multi-scale filter to obtain a filtered i+1th frame motion vector feature map; Inputting the normal vector feature map of the i+1th frame into the third single-scale filter to obtain a filtered normal vector feature map of the i+1th frame; Inputting the filtered motion vector feature map of the i+1th frame and the filtered normal vector feature map of the i+1th frame into the third composite unit to obtain a third composite feature map; Inputting the denoised (i+1)th frame main feature map and the third composite feature map into the third bilinear sampling unit to obtain a third sampling feature map; The third sampling feature map is input into the sixth kernel prediction network to obtain a third image corresponding to the (i+1)th frame of medical image.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images