Image processing method and device, computer equipment and storage medium

Abstract

The invention relates to an image processing method and device, computer equipment and a computer readable storage medium, relates to the technical field of image processing, and can improve the quality of an interpolation intermediate value image. The method comprises the following steps: acquiring a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by performing interpolation between adjacent two-dimensional image frames, and each to-be-adjusted area in the first interpolation intermediate frame sequence is determined; the difference degree between the image visual feature of the to-be-adjusted area and the image visual feature of the corresponding original area in the adjacent two-dimensional image frame is not less than a preset threshold value; and determining a second image rendering feature corresponding to the to-be-adjusted areas according to a first image rendering feature adopted when pixel points in the original areas of the adjacent two-dimensional image frames are rendered, and rendering each to-be-adjusted area according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an image processing device, a computer device, a storage medium and a computer program product. Background Art

[0002] In the data volume rendering of images, by processing and visualizing the voxels in the three-dimensional image, a more realistic and natural visual effect can be presented on the two-dimensional image.

[0003] In the related art, during the image rendering process, in order to display a smooth dynamic picture with a higher frame rate, the number of frames displayed per second is generally increased by interpolation, so that different two-dimensional images can transition smoothly.

[0004] However, when the related art interpolates two consecutive two-dimensional images, abnormal image content and color artifacts often appear in the intermediate frame obtained by interpolation, thereby reducing the quality of the interpolated image. Summary of the invention

[0005] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides an image processing method, comprising:

[0007] Acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0008] Determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold;

[0009] According to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted is determined, and each area to be adjusted is rendered according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

[0010] In one of the embodiments, the first interpolated intermediate frame sequence is obtained by performing color interpolation between adjacent two-dimensional image frames, and the image visual features include pixel values ​​of pixel points;

[0011] The determining of the regions to be adjusted in the first interpolated intermediate frame sequence includes:

[0012] For each interpolated intermediate frame in the first interpolated intermediate frame sequence, determine the pixel value difference between the pixel value of a first pixel point in the interpolated intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame, and determine the degree of difference of the first pixel point according to the pixel value difference statistics result of the first pixel point; the pixel value difference statistics result is obtained by performing statistical processing on the pixel value differences corresponding to the first pixel point;

[0013] Based on each of the first pixel points when the difference degree is not less than a preset threshold, each area to be adjusted in the first interpolated intermediate frame sequence is determined.

[0014] In one embodiment, determining the pixel value difference between the pixel value of the first pixel point in the interpolated intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame includes:

[0015] Determine a first region of interest in the interpolated intermediate frame and a second region of interest in the adjacent two-dimensional image frame corresponding to the first region of interest;

[0016] Obtain pixel value differences between pixel values ​​of each first pixel point in the first region of interest and pixel values ​​of each corresponding second pixel point in the second region of interest.

[0017] In one of the embodiments, determining the second image rendering feature corresponding to the area to be adjusted according to the first image rendering feature used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered includes:

[0018] In the three-dimensional image, voxels associated with the original region are determined, and first image rendering features corresponding to the same voxels are determined from the image rendering features respectively used in the adjacent two-dimensional image frames;

[0019] interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence;

[0020] Based on the correspondence between the interpolation feature sequence and each of the interpolation intermediate frames, the second image rendering feature corresponding to the to-be-adjusted region of each of the interpolation intermediate frames is determined.

[0021] In one embodiment, interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence includes:

[0022] Inputting each first image rendering feature corresponding to the same voxel into an interpolation model;

[0023] The interpolation model interpolates between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence.

[0024] In one embodiment, the determining of the first image rendering feature corresponding to the same voxel from the image rendering features respectively used by the adjacent two-dimensional image frames comprises:

[0025] Determine an associated pixel pair according to the pixel points of each two-dimensional image frame in the original area of ​​the adjacent two-dimensional image frames; the pixel points of the associated pixel pair come from two two-dimensional image frames respectively, and the pixel values ​​of the associated pixel points are obtained by rendering the same voxel in the three-dimensional image according to different image rendering features;

[0026] The image rendering features respectively adopted when the associated pixel pairs are rendered are determined as the first image rendering features in the adjacent two-dimensional image frames.

[0027] In one embodiment, determining the voxels associated with the original region in the three-dimensional image includes:

[0028] Determine the pixel points in the original area;

[0029] Each voxel in the three-dimensional image used for rendering a pixel point in the original region is determined as a voxel associated with the original region.

[0030] In a second aspect, the present application further provides an image processing device, comprising:

[0031] A first intermediate frame sequence acquisition module is used to acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0032] The module for determining the area to be adjusted is used to determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold;

[0033] The second intermediate frame sequence acquisition module is used to determine the second image rendering features corresponding to the area to be adjusted according to the first image rendering features used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, and render each of the areas to be adjusted according to the second image rendering features to obtain a second interpolation intermediate frame sequence.

[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0035] Acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0036] Determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold;

[0037] According to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted is determined, and each area to be adjusted is rendered according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0039] Acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0040] Determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold;

[0041] According to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted is determined, and each area to be adjusted is rendered according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0044] Determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold;

[0045] According to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted is determined, and each area to be adjusted is rendered according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

[0046] The above-mentioned image processing method, apparatus, computer equipment, computer-readable storage medium and computer program product can obtain a first interpolation intermediate frame sequence, which is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features; then, each area to be adjusted in the first interpolation intermediate frame sequence can be determined; the degree of difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding to it in the adjacent two-dimensional image frame is not less than a preset threshold; and then, based on the first image rendering features used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering features corresponding to the area to be adjusted are determined, and each area to be adjusted is rendered according to the second image rendering features to obtain a second interpolation intermediate frame sequence. In this embodiment, on the one hand, a first interpolation intermediate frame sequence is obtained through a related interpolation algorithm, which effectively reduces the computing resource overhead in the process of volume data rendering. On the other hand, by determining the second image rendering features corresponding to the area to be adjusted, and rendering the area to be adjusted in a targeted manner according to the second image rendering features, the visual effects of the areas with obviously abnormal image visual features in the first interpolation intermediate frame sequence can be effectively optimized, thereby significantly improving the image quality of the interpolation intermediate frames while reducing the computing resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 is a schematic flow chart of an image processing method in an embodiment;

[0049] Figure 2 A schematic diagram of an interpolation flow of an interpolation model in an embodiment;

[0050] Figure 3 A rendering mask in one embodiment;

[0051] Figure 4 is a structural block diagram of an image processing device in one embodiment;

[0052] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0053] Figure 6 The figure is a diagram of the internal structure of another computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In order to enable those skilled in the art to better understand the present application, the relevant technology is first introduced below.

[0056] With the development of computers, volumetric data rendering of three-dimensional images is increasingly widely used in a variety of fields, such as video creation, visualization in medical research, and medical image data analysis. In volumetric data rendering of images, by processing and visualizing the voxels in the three-dimensional image, a more realistic and natural visual effect can be presented on the two-dimensional image. For example, by performing volumetric data rendering on three-dimensional medical images, the imaging results of medical images can be converted into a more understandable three-dimensional effect, and then a realistic two-dimensional image of organs and tissues can be presented, so that medical staff can more easily and conveniently analyze more detailed information in the organs and tissues, which helps to improve the effectiveness and safety of medical diagnosis and treatment.

[0057] For 3D data volume rendering, physically based volume rendering algorithms can often produce realistic visual effects, such as volume rendering based on the Monte Carlo method. However, this method requires a large number of samples to obtain converged results. The huge consumption of computing resources means that this volume rendering method can only meet offline rendering requirements, which is greatly limited in clinical applications. In order to meet the real-time rendering requirements, when performing Monte Carlo volume rendering, although real-time rendering can be achieved by sacrificing a certain degree of accuracy, the small number of sampled samples will generate a lot of noise when the volume data interacts, affecting the accuracy of the rendering effect. Therefore, it is often necessary to introduce some post-processing methods for noise reduction.

[0058] In order to better improve the real-time rendering, interpolation technology has gradually been applied. Interpolation technology is a technology that inserts additional frames between existing video frames or rendering frames to increase the frame rate. By interpolating, the real-time rendering can be significantly improved and the smoothness of the picture can be improved. For example, in some medical application scenarios, it is necessary to capture the dynamic movement of tissue structures, such as the dynamic movement of organs or blood vessels in cardiac CT scans or angiography. It can be understood that dynamic medical images at low frame rates may produce visual flickering or jumping, which is not conducive to doctors making correct diagnosis opinions. Therefore, interpolation can be used to provide a higher frame rate to dynamically display medical images. Specifically, the interpolation method can increase the number of frames displayed per second, so that dynamic changes can transition smoothly, reduce visual interference, and provide doctors with longer observation time; at the same time, when interpolating, inserting intermediate frames can also help improve image quality, reduce noise in the image, and improve image clarity and detail display. With the development of artificial intelligence, some interpolation methods will extract information such as albedo and depth through synthetic networks, and directly predict color based on this.

[0059] However, it is found in practice that the intermediate frames obtained by the above interpolation method often have unpredictable behaviors and color artifacts, which reduces the quality of the interpolated graphics.

[0060] Based on this, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0061] In one embodiment, Figure 1 As shown, an image processing method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0062] S101, obtaining a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features.

[0063] In a specific implementation, a three-dimensional image can be obtained, and the image content can be represented by voxels, wherein the voxels can divide the three-dimensional space into small cubes and record corresponding attribute information (such as position, color, density, transparency, etc.), which can be used to describe the appearance and geometric characteristics of the three-dimensional scene.

[0064] In some exemplary embodiments, the three-dimensional image may be a three-dimensional medical image obtained by scanning a scanned object through a medical imaging device, for example, the three-dimensional medical image may be at least one of the following medical images: computed tomography (CT) image, ultrasound image, magnetic resonance imaging (MRI), single-photon emission computed tomography (SPECT), positron emission tomography (PET). Of course, in some other examples, the three-dimensional image may also be a three-dimensional animation image, for example, a three-dimensional image used to construct a virtual three-dimensional scene in virtual reality (VR) and augmented reality (AR) technology.

[0065] After obtaining the three-dimensional image, the three-dimensional image can be rendered using image rendering features to obtain a corresponding two-dimensional image frame.

[0066] Among them, the image rendering features can be understood as the various parameters used in the rendering process of rendering a three-dimensional image into a two-dimensional image. In some exemplary embodiments, the image rendering features may include one or more of material, texture, lighting (such as reflected light and refracted light, etc.), shadow, camera perspective and camera parameters, depth of field, etc.; for example, by rendering according to material features, you can add materials to the three-dimensional model to simulate the surface texture of objects in the real world, such as metal, glass, wood, etc., and by performing texture mapping according to texture features, you can enhance the details and colors of the model. For example, by rendering according to the camera perspective and camera parameters (such as focal length, aperture, exposure time, etc.), you can control the perspective relationship and depth of field effect in the two-dimensional image, so that the rendered image has a stronger sense of three-dimensionality and space.

[0067] A plurality of two-dimensional image frames may be sequentially arranged according to a display timing formation, and adjacent two-dimensional image frames may be determined therefrom according to a timing relationship of the plurality of two-dimensional image frames, wherein adjacent two-dimensional image frames include two temporally adjacent two-dimensional image frames, and each of the adjacent two-dimensional image frames may be rendered according to a corresponding image rendering feature, and the image rendering features used by each of the adjacent two-dimensional image frames may be the same or different. For example, for two-dimensional image frame 1, two-dimensional image frame 2, and two-dimensional image frame 3 sequentially arranged, the image rendering features used by two-dimensional image frame 1 and two-dimensional image frame 2 are the same, and two-dimensional image frame 2 and two-dimensional image frame 3 use different image rendering features.

[0068] After determining the adjacent two-dimensional image frames, interpolation can be performed between the adjacent two-dimensional image frames to obtain multiple interpolation intermediate frames between the adjacent two-dimensional image frames. The multiple intermediate frames constitute an interpolation intermediate frame sequence. To facilitate the distinction from other interpolation intermediate frame sequences in the following text, the interpolation intermediate frame sequence in this step is referred to as a first interpolation intermediate frame sequence, and the interpolation intermediate frames in the first interpolation intermediate frame sequence can also be referred to as first interpolation intermediate frames. Among them, those skilled in the art can select the required interpolation method to obtain the first interpolation intermediate frame sequence according to actual conditions. In this embodiment, the method for obtaining the first interpolation intermediate frame sequence is not limited. For example, the first interpolation intermediate frame sequence can be obtained by an interpolation algorithm such as linear interpolation.

[0069] S102, determining each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the corresponding original area in the adjacent two-dimensional image frame is not less than a preset threshold.

[0070] In a specific implementation, the first interpolated intermediate frame sequence and the adjacent two-dimensional image frames are obtained in different ways, that is, the first interpolated intermediate frame sequence is obtained by interpolation, while the adjacent two-dimensional image frames are obtained by rendering three-dimensional images. Therefore, the visual effects of the first interpolated intermediate frame sequence relative to the adjacent two-dimensional image frames may be abnormal, such as jumps and discontinuities. In some possible embodiments, the intermediate frames can be rendered at a fixed rendering interval. The rendering interval usually requires a trade-off between image quality and computational consumption. A larger rendering interval means lower computational consumption, but it usually also leads to a reduction in image quality.

[0071] In this regard, the present embodiment provides an adaptive rendering strategy. In this step, each area to be adjusted in the first interpolation intermediate frame sequence can be determined, and each area to be adjusted can come from an interpolation intermediate frame in the first interpolation intermediate frame sequence, or can come from multiple interpolation intermediate frames. Specifically, since the interpolation intermediate frame is obtained by interpolating adjacent two-dimensional image frames, each interpolation intermediate frame and the adjacent two-dimensional image frame have corresponding image areas. In other words, for each image area in the interpolation intermediate frame, each two-dimensional image frame in the adjacent two-dimensional image frames has an image area corresponding to the image area of ​​the interpolation intermediate frame, for example, area A in the interpolation intermediate frame corresponds to area A1 of two-dimensional image frame 1 and area A2 of two-dimensional image frame 2 in the adjacent two-dimensional image frames, and the original area of ​​each two-dimensional image frame in the adjacent two-dimensional image frames can be in the same position in the image, that is, the position of area A1 in two-dimensional image frame 1 and the position of area A2 in two-dimensional image frame 2 can be the same.

[0072] In this embodiment, the image area of ​​each interpolated intermediate frame in the first interpolated intermediate frame sequence can be compared with the image area corresponding to each two-dimensional image frame in the adjacent two-dimensional image frames to determine whether there is a significant difference in the image visual features of the image area (such as A) of the interpolated intermediate frame relative to the image visual features of the corresponding image areas (such as A1 and A2) of the adjacent two-dimensional image frames. If the degree of difference in the image visual features is not less than (i.e., greater than or equal to) a preset threshold, it can be determined that the image area is a region to be adjusted. If the degree of difference in the image visual features is less than the preset threshold, it can be determined that the image area is normal and no processing is required. The image visual features can be feature information that can be perceived by the user visually. In some examples, the image visual features can include one or more of the following: color (such as one or more of hue, saturation, and brightness), texture, and edge contour.

[0073] S103, determining a second image rendering feature corresponding to the area to be adjusted according to a first image rendering feature used when rendering pixels in the original area of ​​adjacent two-dimensional image frames, and rendering each area to be adjusted according to the second image rendering feature to obtain a second interpolated intermediate frame sequence.

[0074] After determining the area to be adjusted, it can be determined that there are abnormalities in the image content obtained by interpolation in the area to be adjusted. In this regard, in this embodiment, the image rendering features used when rendering adjacent two-dimensional images can be returned to re-render the area to be adjusted to obtain image content with natural visual effects.

[0075] Specifically, the area to be adjusted corresponds to the original area of ​​the adjacent two-dimensional image frames. For example, the area A of the interpolated intermediate frame mentioned above corresponds to the area A1 of the two-dimensional image frame 1 and the area A2 of the two-dimensional image frame 2 in the adjacent two-dimensional image frames, and the pixel points in the original area of ​​the adjacent two-dimensional image frames are obtained by rendering the three-dimensional image according to the corresponding image rendering features. At the same time, there is a temporal relationship between the intermediate frame and the adjacent two-dimensional image frames, and the image content in the intermediate frame and the adjacent two-dimensional image frame is also spatially related, and the adjacent two-dimensional image frames are obtained by rendering according to the image rendering features.

[0076] Based on this, in this step, the image rendering features used when the pixels in the original areas of adjacent two-dimensional image frames are rendered can be determined, that is, the image rendering features of the original areas of each two-dimensional image frame in the adjacent two-dimensional image frames. For the sake of distinction, this image rendering feature is referred to as the first image rendering feature. Then, the second image rendering features corresponding to the area to be adjusted can be determined based on each first image rendering feature, that is, the image rendering features of the area to be adjusted of the intermediate frame are determined by combining the first image rendering features of the corresponding original areas of the two previous and next two-dimensional image frames. For the sake of distinction, this image rendering feature is referred to as the second image rendering feature. For example, for area A of an interpolated intermediate frame, the first image rendering features of area A1 and the first image rendering features of area A2 can be obtained, and then the second image rendering features of area A can be determined based on the first image rendering features of area A1 and the first image rendering features of area A2.

[0077] After obtaining the second image rendering features corresponding to each area to be adjusted, the area to be adjusted can be rendered according to the second image rendering features corresponding to each area to be adjusted, and a corresponding interpolated intermediate frame sequence can be obtained according to the rendering results. For the sake of distinction, the interpolated intermediate frame sequence is referred to as a second interpolated intermediate frame sequence. In some exemplary embodiments, the area to be adjusted can be rendered according to the second image rendering features corresponding to each area to be adjusted to obtain a rendered image corresponding to the area to be adjusted, and the size of the rendered image matches the size of the area to be adjusted. Then, the rendered image corresponding to each area to be adjusted can replace the corresponding area to be adjusted in the first interpolated intermediate frame sequence. Alternatively, the pixel values ​​of the corresponding areas to be adjusted in the first interpolated intermediate frame sequence can be adjusted according to the pixel values ​​of the rendered images corresponding to each area to be adjusted, thereby obtaining a second interpolated intermediate frame sequence. For example, for two adjacent two-dimensional image frames corresponding to time point 0 and time point 1, the second interpolated intermediate frame sequence can be ,in, , is the interpolated intermediate frame that does not contain the area to be adjusted. It is the interpolated intermediate frame obtained after rendering the area to be adjusted.

[0078] It can be understood that compared with directly predicting the RGB three-dimensional color value of each pixel in the intermediate frame, the image rendered based on the image rendering feature can more accurately simulate the lighting and material effects of the real world, making the obtained image more natural and realistic, but image rendering often takes up a lot of computing resources. If the intermediate frame between adjacent two-dimensional image frames is obtained completely by rendering, the image processing speed is slow; and although image interpolation can reduce the consumption of computing resources, the image obtained by interpolation is usually obtained by inferring the pixel value on the basis of the existing image, and the intermediate frame generated by this method is prone to color artifacts. In this regard, the embodiment of the present application cleverly combines the two methods. First, a first interpolation intermediate frame sequence is obtained by a related interpolation algorithm, and then from the first interpolation intermediate frame sequence, the area to be adjusted in which the image visual features have obvious differences is determined, and then the specific area (that is, the area to be adjusted) in the first interpolation intermediate frame sequence is re-rendered by determining the second image rendering feature corresponding to the obtained area to be adjusted. Thus, the image quality of the interpolation intermediate frame can be effectively improved while reducing the consumption of computing resources. For example, when the area to be adjusted comes from multiple interpolated intermediate frames, when the interval of the interpolated intermediate frames containing the area to be adjusted is 0, it can be understood that there are large changes near the area to be adjusted in each frame after interpolation, so it is necessary to render again frame by frame. When the interval takes other values, it can be understood that the visual effects of some interpolated intermediate frames are abnormal, and the visual effects of some interpolated intermediate frames are normal. Some interpolated intermediate frames can be rendered once at intervals, thereby realizing adaptive intermediate frame rendering in the time domain (interpolated intermediate frames containing the area to be adjusted) and the spatial domain (specific areas in the interpolated intermediate frames, i.e., the area to be adjusted). In some practical examples, the interval value range for re-rendering of interpolated intermediate frames can be [0,25].

[0079] The above-mentioned image processing method can obtain a first interpolation intermediate frame sequence, which is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered to obtain a three-dimensional image according to corresponding image rendering features; then, each area to be adjusted in the first interpolation intermediate frame sequence can be determined; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding to it in the adjacent two-dimensional image frame is not less than a preset threshold; and then according to the first image rendering features used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering features corresponding to the area to be adjusted are determined, and each area to be adjusted is rendered according to the second image rendering features to obtain a second interpolation intermediate frame sequence. In this embodiment, on the one hand, the first interpolation intermediate frame sequence is obtained by a related interpolation algorithm, which effectively reduces the computing resource overhead in the volume data rendering process; on the other hand, by determining the second image rendering features corresponding to the area to be adjusted, and rendering the area to be adjusted in a targeted manner according to the second image rendering features, the visual effect of the area with obviously abnormal image visual features in the first interpolation intermediate frame sequence can be effectively optimized, so that the image quality of the interpolation intermediate frame can be significantly improved while reducing the computing resource consumption.

[0080] In an exemplary embodiment, the first interpolated intermediate frame sequence is obtained by performing color interpolation between adjacent two-dimensional image frames, and the image visual features include pixel values ​​of pixels. Accordingly, in step S102, determining each area to be adjusted in the first interpolated intermediate frame sequence may include the following steps:

[0081] For each interpolation intermediate frame in the first interpolation intermediate frame sequence, the pixel value difference between the pixel value of the first pixel point in the interpolation intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame is determined, and the difference degree of the first pixel point is determined according to the pixel value difference statistics of the first pixel point; the pixel value difference statistics are obtained by performing statistical processing on the pixel value differences corresponding to the first pixel point; based on each first pixel point when the difference degree is not less than a preset threshold, each area to be adjusted in the first interpolation intermediate frame sequence is determined.

[0082] In practical applications, each interpolation intermediate frame in the first interpolation intermediate frame sequence can be traversed. For each interpolation intermediate frame, the pixel points are compared with the pixel points in the adjacent two-dimensional image frame in units of pixels. For the convenience of distinction, the pixel points of the interpolation intermediate frame in the first interpolation intermediate frame sequence are called first pixel points, and the pixel points of each two-dimensional image frame in the adjacent two-dimensional image frames are called second pixel points.

[0083] In the process of comparing pixels, the first pixel point to be compared in the interpolated intermediate frame can be determined. The first pixel point can be all the pixel points of the interpolated intermediate frame or some of the pixel points of the interpolated intermediate frame. For each first pixel point to be compared, the second pixel points corresponding to the first pixel point can be determined from the adjacent two-dimensional image frames. For example, for the first pair of pixel points P, the second pixel point P1 corresponding to the first pair of pixel points P can be determined from the two-dimensional image frame 1 in the adjacent two-dimensional image frames, and the second pixel point P2 corresponding to the first pair of pixel points P can be determined from the two-dimensional image frame 2 in the adjacent two-dimensional image frames.

[0084] Then, the pixel value difference between the pixel value of the first pixel point and the pixel values ​​of each second pixel point corresponding to the first pixel point can be determined. For example, the pixel value difference between the first pair of pixel points P and the second pixel point P1, as well as the pixel value difference between the first pair of pixel points P and another second pixel point P2 can be determined, thereby obtaining the pixel value differences of the first pixel point (that is, two pixel value differences).

[0085] Afterwards, the differences in the pixel values ​​of the first pixel point can be statistically processed to obtain a statistical result of the pixel value difference, and the statistical result of the pixel value difference can be used as the degree of difference between the first pixel point and the corresponding second pixel points. In some examples, the statistical result of the pixel value difference can be an average value. For example, the pixel value difference between the first pair of pixel points P and the second pixel point P1 can be calculated, and the square or absolute value of the pixel value difference can be obtained. At the same time, the pixel value difference between the first pair of pixel points P and the second pixel point P2 can be calculated, and the square or absolute value of the pixel value difference can be obtained. The square or absolute value of the pixel value difference can be used as the pixel value difference. After obtaining the difference between the two pixel values, the average value of the two pixel value differences is calculated and used as the statistical result of the pixel value difference. It can be understood that in addition to obtaining the average value, the statistical processing can also obtain the maximum value, minimum value, etc.

[0086] The difference degree of each first pixel point can be obtained by the above method, and then, the first pixel points are screened according to the preset threshold value, and the first pixel points whose difference degree is not less than the preset threshold value are determined, and the regions to be adjusted in the first interpolation intermediate frame sequence are determined according to the first pixel points. It can be understood that when there are multiple first pixel points, the multiple first pixel points can come from different interpolation intermediate frames in the first interpolation intermediate frame sequence, for example, there are N1 first pixel points in the interpolation intermediate frame 2, and there are N2 first pixel points in the interpolation intermediate frame 3.

[0087] In this embodiment, by determining the pixel value difference between the pixel value of the first pixel point in each interpolated intermediate frame and the pixel value of each corresponding second pixel point in the adjacent two-dimensional image frame, and based on the statistical results of the pixel value difference of the first pixel point, determining the degree of difference of the first pixel point and comparing it with a preset threshold, it is possible to quickly identify each area to be adjusted in the first interpolated intermediate frame sequence based on the pixel value of the image, thereby effectively improving the image processing efficiency.

[0088] In one embodiment, determining the pixel value difference between the pixel value of a first pixel point in the interpolated intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame may include the following steps:

[0089] Determine a first region of interest in the interpolated intermediate frame and a second region of interest corresponding to the first region of interest in an adjacent two-dimensional image frame; obtain pixel value differences between pixel values ​​of each first pixel point in the first region of interest and pixel values ​​of each second pixel point corresponding thereto in the second region of interest.

[0090] In some exemplary embodiments, all first pixel points in the interpolated intermediate frame can be compared with the corresponding second pixel points in the adjacent two-dimensional image frame, that is, pixel comparison of the entire image can be performed. This method can achieve refined identification of the area to be adjusted, but it consumes a lot of computing resources. In this regard, in this embodiment, the region of interest in the interpolated intermediate frame can be determined. For the sake of distinction, the region of interest is called the first region of interest.

[0091] Among them, the region of interest can be the area that the user is concerned about. Taking medical images as an example, medical images may contain multiple tissues, organs, and structures. The target tissue organs that the user is concerned about, such as the heart or liver, can be determined from the multiple tissues, organs, and structures through an image segmentation algorithm, and the area where the target tissue organ is located in the interpolated intermediate frame is determined as the first region of interest; or, the area where the image visual feature abnormalities are more likely to occur can be determined as the region of interest. For example, interpolation errors are often concentrated on relatively small and fast-moving objects, while almost static or slowly moving background information can often be correctly interpolated. In this regard, the foreground and background of the interpolated intermediate frame can be identified, and the foreground content can be determined as the first region of interest.

[0092] After determining the first region of interest, each second region of interest corresponding to the first region of interest may be determined in each two-dimensional image frame of adjacent two-dimensional image frames. For example, after determining the first region of interest C in the interpolated intermediate frame, the second region of interest C1 corresponding to the first region of interest C may be determined in two-dimensional image frame 1 in the adjacent two-dimensional image frames, and the second region of interest C2 corresponding to the first region of interest C may be determined in two-dimensional image frame 2. Subsequently, the pixel value difference between the pixel value of each first pixel point in the first region of interest and the pixel value of each second pixel point corresponding thereto in the second region of interest may be obtained. The comparison process of the pixel points may refer to the aforementioned embodiment, which is not described in detail here.

[0093] In this embodiment, by obtaining the pixel value difference between the pixel values ​​of each first pixel point in the first region of interest and the pixel value of each corresponding second pixel point in the second region of interest, the area to be adjusted in the region of interest of the user can be specifically identified, which helps to achieve a balance between image quality and computational consumption.

[0094] In one embodiment, in step S103, determining the second image rendering feature corresponding to the area to be adjusted according to the first image rendering feature used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered may include the following steps:

[0095] In a three-dimensional image, voxels associated with an original area are determined, and first image rendering features corresponding to the same voxel are determined from the image rendering features respectively adopted by adjacent two-dimensional image frames; interpolation is performed between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence; based on the correspondence between the interpolation feature sequence and each interpolation intermediate frame, the second image rendering feature corresponding to the area to be adjusted of each interpolation intermediate frame is determined.

[0096] In practical applications, the image content of each two-dimensional image frame is obtained by rendering the voxels in the three-dimensional image frame using the corresponding image rendering features. After determining the original region in the adjacent two-dimensional image frame, the voxels associated with the original region can be determined in the three-dimensional image, wherein the voxels associated with the original region refer to the voxels in the three-dimensional image that need to be used (or involved) when rendering the image content of the original region.

[0097] Then, each first image rendering feature corresponding to the same voxel can be determined from the image rendering features respectively used by the adjacent two-dimensional image frames. Specifically, the change of the image content in the adjacent two-dimensional image frames often comes from the change of the image rendering feature. When the three-dimensional image remains unchanged, by changing the image rendering feature, multiple two-dimensional image frames with different image spatial information can be obtained, such as multiple two-dimensional image frames with changed light illumination angle, object shadow or visibility. Based on this, after determining the voxels associated with the original area, each first image rendering feature corresponding to the same voxel can be determined from the image rendering features respectively used by the adjacent two-dimensional image frames. For example, for the same voxel in the three-dimensional image, the image content 1 of the two-dimensional image frame 1 in the adjacent two-dimensional image frame is obtained by rendering using the image rendering feature 1, and the image content 2 of the two-dimensional image frame 2 in the adjacent two-dimensional image frame is obtained by rendering using the image rendering feature 2. When rendering the two-dimensional image frame, the corresponding relationship between the voxel, the pixel point and the image rendering feature can be recorded. Then, after determining the voxels associated with the original area, each first image rendering feature corresponding to the same voxel can be determined from the image rendering features respectively used by the adjacent two-dimensional image frames.

[0098] After obtaining each first image rendering feature corresponding to the same voxel, interpolation can be performed between each first image rendering feature to obtain an interpolation feature sequence. The interpolation feature sequence includes a plurality of interpolated features, and the feature dimension of the interpolated features is the same as the feature dimension of the first image rendering features. For example, if each first image rendering feature is represented by a vector, the dimension of the vector may be N, indicating that it contains N different first image rendering features. Accordingly, when the interpolated features are represented by a vector, their dimension is also N. Furthermore, the number of features included in the interpolation feature sequence is the same as the number of interpolated intermediate frames in the first interpolated intermediate frame sequence. For example, if the first interpolated intermediate frame sequence contains N interpolated intermediate frames, the interpolation feature sequence includes N features.

[0099] Furthermore, the second image rendering feature corresponding to the area to be adjusted of each interpolated intermediate frame can be determined based on the correspondence between the interpolated feature sequence and each interpolated intermediate frame in the first interpolated intermediate frame sequence. For example, each feature in the interpolated feature sequence can be arranged in sequence according to the time sequence, and each interpolated intermediate frame in the first interpolated intermediate frame sequence is also arranged in sequence according to the time sequence, then it can be determined that there is a correspondence between the features with the same order in the two sequences and the interpolated intermediate frames, and the area to be adjusted in the interpolated intermediate frame can be rendered later according to the correspondence.

[0100] In this embodiment, by determining the first image rendering features corresponding to the same voxel from the image rendering features used by adjacent two-dimensional image frames, and interpolating between the first image rendering features, a smoother and more coherent second image rendering feature can be obtained. Subsequently, the area to be adjusted is rendered based on the second image rendering feature, which helps to reduce anomalies such as color artifacts generated during interpolation, makes the transition of the intermediate frame sequence more natural, and improves the image quality of the interpolated intermediate frames.

[0101] In one embodiment, interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence may include the following steps:

[0102] The first image rendering features corresponding to the same voxel are input into the interpolation model; the interpolation model interpolates between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence.

[0103] In a specific implementation, the interpolation model can be pre-trained. In some exemplary embodiments, the neural network can be trained through supervised training, for example, a grid neural network (gridnet) can be trained. By performing model training, the interpolation model can learn the interpolation law and perform appropriate interpolation between multiple image rendering features. At the same time, the interpolation model can be trained to obtain pre-trained weights corresponding to the interpolation law. In some examples, the pre-trained weights can include optical flow information corresponding to adjacent two-dimensional image frames in different scenes. After obtaining each first image rendering feature corresponding to the same voxel, the interpolation model can interpolate between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence.

[0104] For example, Figure 2 As shown, for adjacent video frames I1 and I2, the pixel difference between each pixel in each interpolated intermediate frame and the corresponding pixel in video frame I1 and video frame I2 can be determined, thereby obtaining a matrix consisting of multiple pixel difference sizes corresponding to each interpolated intermediate frame, and the first image rendering features corresponding to each of video frames I1 and I2 can also be obtained, and then the multiple matrices and the first image rendering features corresponding to each of video frames I1 and I2 are input into the interpolation model, and the interpolation model can obtain an interpolation feature sequence based on the input content and pre-trained weights. In addition, the matrix consisting of multiple pixel difference sizes can be input into the VGG network for binarization to obtain a binarized matrix, wherein if the element in the matrix is ​​0, no rendering is required, and if the element in the matrix is ​​1, the pixel corresponding to the element in the interpolated intermediate frame can be rendered. In some examples, a corresponding rendering mask can be generated based on the binarized matrix, such as Figure 3As shown, the rendering mask identifies the part of the current interpolation intermediate frame that needs to be rendered. Then, the rendering mask of each interpolation intermediate frame and the interpolation feature sequence output by the model can be combined to perform rendering to obtain a second interpolation intermediate frame sequence.

[0105] In some examples, for ease of understanding, the interpolation process of the interpolation model can be regarded as linear interpolation, so that the second interpolation intermediate frame is only a linear combination of adjacent two-dimensional image frames. Since the color of the output pixel is limited to the convex hull of the input color, the generation of color artifacts can be avoided.

[0106] For example, the interpolation model can estimate the feature representation vector (that is, each feature in the interpolated feature sequence) based on the attention mechanism. The parameters are used in the linear interpolation of this example. and parameters ,in, For the size The estimated dimension of the image region (also called the kernel function size, which matches the size of the region to be adjusted) is The characteristic value of , D is the dimension of the image rendering feature, corresponding to the number of rendering channels. Each rendering channel can render related image information according to an image rendering feature. is the offset value corresponding to the linear function, parameter and parameters The first image rendering features are used to perform linear interpolation to obtain an interpolation feature sequence.

[0107] In this embodiment, the interpolation model interpolates between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence. The interpolation model can combine the prior information learned in the past to more accurately interpolate between the first image rendering features, thereby improving the interpolation accuracy of the interpolation feature sequence. At the same time, the interpolation feature sequence is obtained by sharing the pre-trained weights of the interpolation model, so that parameters can be shared in the process of interpolating all image rendering features, thereby avoiding abrupt content in the image and improving the robustness of the rendered video interpolation.

[0108] In one embodiment, determining the first image rendering feature corresponding to the same voxel from the image rendering features respectively used by adjacent two-dimensional image frames may include the following steps:

[0109] According to the pixel points of each two-dimensional image frame in the original area of ​​adjacent two-dimensional image frames, associated pixel pairs are determined; the pixel points of the associated pixel pairs come from the two two-dimensional image frames respectively, and the pixel values ​​of the associated pixel points are obtained by rendering the same voxels in the three-dimensional image according to different image rendering features; the image rendering features applied when the associated pixel pairs are rendered are determined as the first image rendering features in the adjacent two-dimensional image frames.

[0110] In practical applications, the pixels of each two-dimensional image frame in the original area of ​​adjacent two-dimensional image frames can be matched to determine the associated pixel pairs. The associated pixel pairs contain two pixels, which come from two two-dimensional image frames in the adjacent two-dimensional image frames. At the same time, the pixel values ​​of the two pixels in the associated pixel pairs are obtained by rendering the same voxel in the three-dimensional image according to different image rendering features. In other words, although the two pixels in the associated pixel pairs come from different two-dimensional image frames, the two pixels actually correspond to the same voxel in the three-dimensional space. In some examples, the associated pixel pairs can be determined by a matching algorithm, or by the correspondence between pixels and voxels pre-stored during rendering.

[0111] Furthermore, the image rendering features used when the associated pixel pairs are rendered can be determined as the first image rendering features in adjacent two-dimensional image frames. For example, for the associated pixel pair (p1, p2), the image rendering features corresponding to p1 and the image rendering features corresponding to p2 can be determined as the first image rendering features in adjacent two-dimensional image frames.

[0112] In this embodiment, by determining associated pixel points and determining the image rendering features used by each associated pixel pair when rendered as the first image rendering features in adjacent two-dimensional image frames, the first image rendering features corresponding to the same voxel can be quickly found through the correspondence between pixel points and voxels.

[0113] In one embodiment, in a three-dimensional image, determining voxels associated with an original region may include the following steps:

[0114] Pixels in the original region are determined; and each voxel in the three-dimensional image used to render the pixels in the original region is determined as a voxel associated with the original region.

[0115] In a specific implementation, the pixel points in the original area can be determined in at least one of the adjacent two-dimensional image frames. When the original areas of each of the adjacent two-dimensional image frames are the same, only one of the two-dimensional image frames can be selected and the pixel points in the original area of ​​the two-dimensional image frame can be determined. Then, in the three-dimensional image, each voxel used to render the pixel points in the original area can be determined, that is, according to the corresponding relationship between the pixels in the two-dimensional image frame and the voxels in the three-dimensional image, the voxels corresponding to the pixel points in the original area are searched, and then each determined voxel can be determined as a voxel associated with the original area.

[0116] In this embodiment, by determining each voxel in the three-dimensional image used to render the pixel points in the original area as the voxel associated with the original area, the voxels related to the area to be adjusted can be accurately matched, which helps to accurately locate the matching first image rendering features subsequently, thereby improving the accuracy of the second image rendering features, reducing errors, and improving the rendering effect of the interpolated intermediate frames.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0119] In an exemplary embodiment, Figure 4 As shown, an image processing device is provided, comprising:

[0120] A first intermediate frame sequence acquisition module 401 is used to acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features;

[0121] The to-be-adjusted region determination module 402 is used to determine each to-be-adjusted region in the first interpolation intermediate frame sequence; the difference between the image visual features of the to-be-adjusted region and the image visual features of the corresponding original region in the adjacent two-dimensional image frame is not less than a preset threshold;

[0122] The second intermediate frame sequence acquisition module 403 is used to determine the second image rendering features corresponding to the area to be adjusted based on the first image rendering features used when the pixels in the original area of ​​the adjacent two-dimensional image frame are rendered, and render each area to be adjusted according to the second image rendering features to obtain a second interpolation intermediate frame sequence.

[0123] In one embodiment, the first interpolated intermediate frame sequence is obtained by performing color interpolation between adjacent two-dimensional image frames, and the image visual features include pixel values ​​of pixel points;

[0124] The to-be-adjusted area determination module 402 is used to:

[0125] For each interpolated intermediate frame in the first interpolated intermediate frame sequence, determine the pixel value difference between the pixel value of a first pixel point in the interpolated intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame, and determine the degree of difference of the first pixel point according to the pixel value difference statistics result of the first pixel point; the pixel value difference statistics result is obtained by performing statistical processing on the pixel value differences corresponding to the first pixel point;

[0126] Based on each of the first pixel points when the difference degree is not less than a preset threshold, each area to be adjusted in the first interpolated intermediate frame sequence is determined.

[0127] In one embodiment, the to-be-adjusted area determination module 402 is configured to:

[0128] Determine a first region of interest in the interpolated intermediate frame and a second region of interest in the adjacent two-dimensional image frame corresponding to the first region of interest;

[0129] Obtain pixel value differences between pixel values ​​of each first pixel point in the first region of interest and pixel values ​​of each corresponding second pixel point in the second region of interest.

[0130] In one embodiment, the second intermediate frame sequence acquisition module 403 is used to:

[0131] In the three-dimensional image, voxels associated with the original region are determined, and first image rendering features corresponding to the same voxels are determined from the image rendering features respectively used in the adjacent two-dimensional image frames;

[0132] interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence;

[0133] Based on the correspondence between the interpolation feature sequence and each of the interpolation intermediate frames, the second image rendering feature corresponding to the to-be-adjusted region of each of the interpolation intermediate frames is determined.

[0134] In one embodiment, the second intermediate frame sequence acquisition module 403 is used to:

[0135] Inputting each first image rendering feature corresponding to the same voxel into an interpolation model;

[0136] The interpolation model interpolates between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence.

[0137] In one embodiment, the second intermediate frame sequence acquisition module 403 is used to:

[0138] Determine an associated pixel pair according to the pixel points of each two-dimensional image frame in the original area of ​​the adjacent two-dimensional image frames; the pixel points of the associated pixel pair come from two two-dimensional image frames respectively, and the pixel values ​​of the associated pixel points are obtained by rendering the same voxel in the three-dimensional image according to different image rendering features;

[0139] The image rendering features respectively adopted when the associated pixel pairs are rendered are determined as the first image rendering features in the adjacent two-dimensional image frames.

[0140] In one embodiment, the second intermediate frame sequence acquisition module 403 is used to:

[0141] Determine the pixel points in the original area;

[0142] Each voxel in the three-dimensional image used for rendering a pixel point in the original region is determined as a voxel associated with the original region.

[0143] Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0144] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store three-dimensional images and two-dimensional image frames. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0145] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0146] Those skilled in the art will understand that Figure 5 and Figure 6The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0148] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0149] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0150] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0151] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features; Determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold; According to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted is determined, and each area to be adjusted is rendered according to the second image rendering feature to obtain a second interpolation intermediate frame sequence.

2. The method according to claim 1, characterized in that The first interpolation intermediate frame sequence is obtained by performing color interpolation between adjacent two-dimensional image frames, and the image visual features include pixel values ​​of pixel points; The determining of the regions to be adjusted in the first interpolated intermediate frame sequence includes: For each interpolated intermediate frame in the first interpolated intermediate frame sequence, determine the pixel value difference between the pixel value of a first pixel point in the interpolated intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame, and determine the degree of difference of the first pixel point according to the pixel value difference statistics result of the first pixel point; the pixel value difference statistics result is obtained by performing statistical processing on the pixel value differences corresponding to the first pixel point; Based on each of the first pixel points when the difference degree is not less than a preset threshold, each area to be adjusted in the first interpolated intermediate frame sequence is determined.

3. The method according to claim 2, characterized in that The determining of the pixel value difference between the pixel value of the first pixel point in the interpolation intermediate frame and the pixel values ​​of each corresponding second pixel point in the adjacent two-dimensional image frame includes: Determine a first region of interest in the interpolated intermediate frame and a second region of interest in the adjacent two-dimensional image frame corresponding to the first region of interest; Obtain pixel value differences between pixel values ​​of each first pixel point in the first region of interest and pixel values ​​of each corresponding second pixel point in the second region of interest.

4. The method according to claim 1, characterized in that: The determining, according to the first image rendering feature adopted when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, the second image rendering feature corresponding to the area to be adjusted includes: In the three-dimensional image, voxels associated with the original region are determined, and first image rendering features corresponding to the same voxels are determined from the image rendering features respectively used in the adjacent two-dimensional image frames; interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence; Based on the correspondence between the interpolation feature sequence and each of the interpolation intermediate frames, the second image rendering feature corresponding to the to-be-adjusted region of each of the interpolation intermediate frames is determined.

5. The method according to claim 4, characterized in that The interpolating between the first image rendering features corresponding to the same voxel to obtain an interpolation feature sequence comprises: Inputting each first image rendering feature corresponding to the same voxel into an interpolation model; The interpolation model interpolates between the first image rendering features according to the pre-trained weights to obtain an interpolation feature sequence.

6. The method according to claim 4, characterized in that The determining, from the image rendering features respectively used by the adjacent two-dimensional image frames, a first image rendering feature corresponding to the same voxel comprises: Determine an associated pixel pair according to the pixel points of each two-dimensional image frame in the original area of ​​the adjacent two-dimensional image frames; the pixel points of the associated pixel pair come from two two-dimensional image frames respectively, and the pixel values ​​of the associated pixel points are obtained by rendering the same voxel in the three-dimensional image according to different image rendering features; The image rendering features respectively adopted when the associated pixel pairs are rendered are determined as the first image rendering features in the adjacent two-dimensional image frames.

7. The method according to claim 4, characterized in that Determining, in the three-dimensional image, voxels associated with the original region comprises: Determine the pixel points in the original area; Each voxel in the three-dimensional image used for rendering a pixel point in the original region is determined as a voxel associated with the original region.

8. An image processing device, characterized in that: The device comprises: A first intermediate frame sequence acquisition module is used to acquire a first interpolation intermediate frame sequence; the first interpolation intermediate frame sequence is obtained by interpolating between adjacent two-dimensional image frames, and the adjacent two-dimensional image frames are rendered into a three-dimensional image according to corresponding image rendering features; The module for determining the area to be adjusted is used to determine each area to be adjusted in the first interpolation intermediate frame sequence; the difference between the image visual features of the area to be adjusted and the image visual features of the original area corresponding thereto in the adjacent two-dimensional image frame is not less than a preset threshold; The second intermediate frame sequence acquisition module is used to determine the second image rendering features corresponding to the area to be adjusted according to the first image rendering features used when the pixel points in the original area of ​​the adjacent two-dimensional image frame are rendered, and render each of the areas to be adjusted according to the second image rendering features to obtain a second interpolation intermediate frame sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images