Image processing method and device

Abstract

The present disclosure relates to an image processing method and device. An image pyramid is created according to a feature image corresponding to a target, the image pyramid includes multiple layers, each layer is an image; a target layer is determined from each layer of the image pyramid according to the distance between the first three-dimensional coordinate of the target and the camera origin; a feature value is retrieved using the image corresponding to the target layer in the image pyramid to obtain a retrieved feature value; the retrieved feature value is input into a preset coordinate network to predict the target body density and target feature corresponding to the target; the target feature and the first light direction corresponding to the target are input into a preset direction network to predict the target color corresponding to the first three-dimensional coordinate; the target is rendered according to the target color and target body density to obtain a rendered image corresponding to the target. Anti-aliasing is achieved, and training and rendering are fast and accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] Neural Radiance Field (NeRF)-based methods are often used in new perspective generation and reconstruction tasks in image processing. Figure 1 The original NeRF shown in the figure uses a multilayer perceptron (MLP) network to regress the opacity and density of spatial points, which results in a very large amount of computation and cannot be trained and rendered quickly. Grid-based methods (such as DVGO (Deep Visual Grid CNN)) or Hash-based methods (such as Instant-ngp) can effectively improve the network speed, but cannot handle aliasing caused by multiple resolutions. Aliasing is caused by high resolution projected onto low resolution, resulting in jagged edges, such as Figure 2 As shown in Figure 1. Since each pixel of an image can actually be regarded as a light cone, the farther away, the lower the resolution. When distant objects are projected onto the image, aliasing will occur. In related technologies, Mip-NeRF first proposed to approximate the points in the light cone as Gaussians, such as Figure 3 As shown in the figure, compared with NeRF, Mip-NeRF directly uses MLP to regress opacity and color from the mean and variance of the input Gaussian. However, although Mip-NeRF solves the aliasing problem, it cannot be extended to the Grid-based method, which means it cannot be accelerated. How to propose an anti-aliasing solution that can be used for the Grid-based method, so as to obtain the improved effect brought by anti-aliasing and enable fast training and rendering, is a technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of this, the present disclosure proposes an image processing method and device.

[0004] According to one aspect of the present disclosure, there is provided an image processing method, the method comprising:

[0005] Inputting a first three-dimensional coordinate of the target and a corresponding first light direction into a pre-constructed neural radiation field model, and predicting a first eigenvalue corresponding to the first three-dimensional coordinate;

[0006] Creating an image pyramid according to the feature image corresponding to the target, wherein the image pyramid includes a plurality of groups, each group includes at least one layer, and each layer is an image;

[0007] Determining a target layer from each layer of the image pyramid according to the distance between the first three-dimensional coordinate and the camera origin;

[0008] Performing feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain a retrieval feature value;

[0009] Inputting the retrieved feature value into a preset coordinate network to predict the target volume density and target features corresponding to the target;

[0010] Inputting the target feature and the first light direction into a preset direction network to predict a target color corresponding to the first three-dimensional coordinate;

[0011] The target is rendered according to the target color and the target volume density to obtain a rendered image corresponding to the target.

[0012] In a possible implementation manner, images at different layers in the image pyramid correspond to different Gaussian radii, and a target layer is determined from each layer of the image pyramid according to a distance between the first three-dimensional coordinate of the target and a camera origin, including:

[0013] Determine a corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera;

[0014] The difference between the Gaussian radius of each layer in the image pyramid and the target radius is determined, and two layers with the smallest difference among the layers in the image pyramid are determined as target layers.

[0015] In a possible implementation, performing feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain the retrieval feature value includes:

[0016] Projecting the 3D cone corresponding to the target onto the image corresponding to each target layer, and obtaining corresponding plane triangles on the image corresponding to each layer in the target layer;

[0017] Perform feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain a retrieval feature value;

[0018] The retrieval feature value is determined by calculating the retrieval feature value through three-dimensional interpolation.

[0019] In a possible implementation, performing feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain the retrieval feature value includes:

[0020] Extracting feature values ​​of the area where the plane triangle is located in the image corresponding to each target layer to obtain multiple feature values ​​to be retrieved;

[0021] A matching search is performed on multiple feature values ​​to be searched to obtain multiple search feature values ​​that match the target.

[0022] In a possible implementation, the image pyramid includes a first image pyramid, the first image pyramid includes a plurality of groups, each group includes multiple layers, and the image pyramid is created according to a feature image corresponding to the target, including:

[0023] Extracting features of the original image where the target is located through three planes to obtain a first feature image;

[0024] The first image pyramid is created according to the first feature image.

[0025] In a possible implementation, the image pyramid includes a second image pyramid, the second image pyramid includes a plurality of groups, each group includes one or more layers, and the image pyramid is created according to a feature image corresponding to the target, including:

[0026] Extracting features from the original image where the target is located to obtain a second feature image;

[0027] generating an underlying image according to the second feature image;

[0028] Rotating the underlying image according to a preset angle to obtain a rotated image, wherein the preset angle is determined according to the number of angle sampling;

[0029] The second image pyramid is created by repeatedly iterating according to the rotated image and the Gaussian convolution kernel size corresponding to each group.

[0030] In a possible implementation, there is a first proportional relationship between the length of the Gaussian convolution kernel corresponding to the current group and the length of the Gaussian convolution kernel corresponding to the previous group, and there is a second proportional relationship between the width of the Gaussian convolution kernel corresponding to the current group and the width of the Gaussian convolution kernel corresponding to the previous group.

[0031] According to another aspect of the present disclosure, there is provided an image processing device, the device comprising:

[0032] A pyramid creation module, used to create an image pyramid according to a feature image corresponding to the target, wherein the image pyramid includes a plurality of layers, each layer being an image;

[0033] a target layer determination module, configured to determine a target layer from each layer of the image pyramid according to a distance between the first three-dimensional coordinates of the target and a camera origin;

[0034] A feature value retrieval module, used to perform feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain a retrieval feature value;

[0035] A target prediction module, used for inputting the retrieved feature value into a preset coordinate network to predict the target volume density and target features corresponding to the target;

[0036] A color prediction module, used for inputting the target feature and the first light direction corresponding to the target into a preset direction network, and predicting the target color corresponding to the first three-dimensional coordinate;

[0037] A rendering module is used to render the target according to the target color and the target volume density to obtain a rendering image corresponding to the target.

[0038] In a possible implementation manner, the Gaussian radii corresponding to images at different layers in the image pyramid are different, and the target layer determination module includes:

[0039] A radius determination submodule, used to determine a corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera;

[0040] The layer determination submodule is used to determine the difference between the Gaussian radius of each layer in the image pyramid and the target radius, and determine the two layers with the smallest difference among the layers in the image pyramid as target layers.

[0041] In a possible implementation, the feature value retrieval module includes:

[0042] A projection submodule, used for projecting the 3D cone corresponding to the target onto the image corresponding to each target layer, and obtaining corresponding plane triangles on the image corresponding to each layer in the target layer;

[0043] A retrieval submodule, used for performing feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain a retrieval feature value;

[0044] The calculation submodule is used to determine the retrieval feature value by calculating through three-property interpolation according to the retrieval feature value.

[0045] In a possible implementation, the image pyramid includes a first image pyramid, the first image pyramid includes a plurality of groups, each group includes multiple layers, and the pyramid creation module includes:

[0046] A first feature extraction submodule is used to extract features of the original image where the target is located through three planes to obtain a first feature image;

[0047] The first creation submodule is used to create a first image pyramid according to the first feature image.

[0048] In a possible implementation, the image pyramid includes a second image pyramid, the second image pyramid includes a plurality of groups, each group includes one or more layers, and the pyramid creation module includes:

[0049] A second feature extraction submodule is used to extract features from the original image where the target is located to obtain a second feature image;

[0050] An image generation submodule, used for generating an underlying image according to the second feature image;

[0051] A rotation submodule, used to rotate the bottom layer image according to a preset angle to obtain a rotated image, wherein the preset angle is determined according to the number of angle sampling;

[0052] The second creation submodule is used to repeatedly iterate according to the rotated image and the Gaussian convolution kernel size corresponding to each group to create a second image pyramid.

[0053] In a possible implementation, there is a first proportional relationship between the length of the Gaussian convolution kernel corresponding to the current group and the length of the Gaussian convolution kernel corresponding to the previous group, and there is a second proportional relationship between the width of the Gaussian convolution kernel corresponding to the current group and the width of the Gaussian convolution kernel corresponding to the previous group.

[0054] According to another aspect of the present disclosure, there is provided an image processing device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0055] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0056] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0057] The image processing method and device provided by the embodiment of the present disclosure create an image pyramid according to the feature image corresponding to the target, wherein the image pyramid includes multiple layers, each layer being an image; the target layer is determined from each layer of the image pyramid according to the distance between the first three-dimensional coordinate of the target and the camera origin; the image corresponding to the target layer in the image pyramid is used to perform feature value retrieval to obtain retrieval feature values; the retrieval feature values ​​are input into a preset coordinate network to predict the target body density and target features corresponding to the target; the target features and the first light direction corresponding to the target are input into a preset direction network to predict the target color corresponding to the first three-dimensional coordinate; the target is rendered according to the target color and the target body density to obtain a rendered image corresponding to the target. It is possible to ensure fast and accurate training and rendering while anti-aliasing.

[0058] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0060] Figure 1 A schematic diagram showing rendering based on NeRF is shown.

[0061] Figure 2 The aliasing phenomenon in the related art and the schematic diagram of the implementation of anti-aliasing are shown.

[0062] Figure 3 Schematic diagrams of Mip-NeRF and NeRF are shown.

[0063] Figure 4 A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0064] Figure 5 A schematic flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0065] Figure 6 A schematic flow chart of step S102 in the image processing method according to an embodiment of the present disclosure is shown.

[0066] Figure 7 A schematic diagram of sampling point distribution in an image processing method according to an embodiment of the present disclosure is shown.

[0067] Figure 8-Figure 10 A schematic diagram showing the construction of a second image pyramid in an image processing method according to an embodiment of the present disclosure is shown.

[0068] Fig.11 A schematic diagram of target layer determination in an image processing method according to an embodiment of the present disclosure is shown.

[0069] Fig.12 A schematic diagram showing the prediction of target density and target features in an image processing method according to an embodiment of the present disclosure.

[0070] Fig.13 A block diagram of an image processing apparatus according to an embodiment of the present disclosure is shown.

[0071] Fig.14 is a block diagram of a device 1900 for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0072] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0073] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0074] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0075] like Figure 1 As shown, the process of rendering using the neural radiation field model includes: inputting the three-dimensional coordinates of the target and the corresponding light direction into the pre-constructed neural radiation field model, and predicting the eigenvalues ​​(feature) and volume density (density) corresponding to the three-dimensional coordinates. The initial neural radiation field model can be pre-constructed, and the initial neural radiation field model can be trained using an atlas of known poses to obtain a trained neural radiation field model. In the process of training the neural radiation field model, a network f can be used to fit the mapping relationship between the corresponding 5D coordinates (x, y, z, θ, φ) of a certain spatial point (x, y, z) to RGB color, that is, RGBσ=f(x, y, z, θ, φ). As Figure 1As shown in the figure, the trained neural radiation field model can take the three-dimensional coordinates and light direction of the target as input, and output the eigenvalue and volume density of the target. Then, based on the eigenvalue, the color of the rendered image or other image processing tasks can be solved, and finally the target is rendered according to the solved color and the previous volume density. However, the large amount of calculation makes the rendering speed very slow.

[0076] In order to solve the above technical problems, the embodiments of the present disclosure provide an image processing method and device, which can ensure fast and accurate training and rendering while anti-aliasing.

[0077] like Figure 4 As shown, the image processing method may include steps S101 to S106. The method may be applied to application scenarios such as real-time rendering, virtual reality (VR), augmented reality (AR), etc., and the present disclosure does not limit this. Figure 4 , Figure 5 An explanation of the image processing method is given.

[0078] In step S101, an image pyramid is created according to a feature image corresponding to a target, wherein the image pyramid includes a plurality of levels, each of which is an image. In some embodiments, the image pyramid may include a plurality of octaves, each of which includes at least one layer, each of which is an image. The Gaussian radii corresponding to images at different levels in the image pyramid are different.

[0079] In this embodiment, the target may be a target object to be rendered, the target corresponds to a point, line or surface of the target object, and the target object may be an entity to be rendered, which is not limited in the present disclosure. The created image pyramid may be a Gaussian pyramid.

[0080] In some embodiments, step S101 may create an image pyramid in the following two ways:

[0081] Method 1:

[0082] In the first approach, the image pyramid is the first image pyramid. In this case, the first image pyramid may include multiple groups, and each group includes multiple layers.

[0083] Then, the implementation method of creating the first image pyramid may be: extracting features of the original image where the target is located through three planes to obtain a first feature image; and creating the following image pyramid according to the first feature image: Figure 6 The first image pyramid is shown in FIG. The first image pyramid is implemented by a Gaussian convolution kernel downsampling operation. Figure 6 The first image pyramid in includes 5 groups, and the Gaussian radii corresponding to different layers are σ, 2σ, etc.

[0084] In this embodiment, the three planes may be multiple planes for extracting features of the target. In some embodiments, these "planes" may include: a color plane, which analyzes the color distribution of the image and extracts color features, such as a color histogram, a color moment, etc. A texture plane, which analyzes the texture information of the image and extracts local texture features, such as local binary patterns (LBP), gray-level co-occurrence matrix (GLCM), etc. A shape plane, which analyzes the shape structure of the image and extracts geometric features of edges, contours or shapes, such as using edge detection algorithms (such as Sobel, Canny operators) or Hough transforms.

[0085] Method 2:

[0086] In the second approach, the image pyramid is the second image pyramid. In this case, the second image pyramid may include multiple groups, each of which includes one or more layers.

[0087] Then, the implementation method of creating the second image pyramid can be: extracting features from the original image where the target is located to obtain a second feature image; generating an underlying image based on the second feature image; rotating the underlying image according to a preset angle to obtain a rotated image, where the preset angle is determined according to the number of angle samplings; and repeatedly iterating according to the rotated image and the Gaussian convolution kernel size corresponding to each group to create a second image pyramid.

[0088] In some embodiments, in the process of creating the second image pyramid, there is a first proportional relationship between the length of the Gaussian convolution kernel corresponding to the current group and the length of the Gaussian convolution kernel corresponding to the previous group, and there is a second proportional relationship between the width of the Gaussian convolution kernel corresponding to the current group and the width of the Gaussian convolution kernel corresponding to the previous group. The first proportional relationship and the second proportional relationship may be the same or different. For example, the length of the Gaussian convolution kernel corresponding to the current group is half of the length of the Gaussian convolution kernel corresponding to the previous group, and the width of the Gaussian convolution kernel corresponding to the current group is half of the width of the Gaussian convolution kernel corresponding to the previous group. The first proportional relationship and the second proportional relationship may be set according to actual needs, and the present disclosure does not limit this.

[0089] Among them, Figure 7 As shown, due to the different sampling densities of spatial points, we hope to sample more points near the surface, and the light cone will be approximated as an ellipsoid with greater anisotropy. For this reason, we create a second image pyramid with anisotropy.

[0090] For example, the ellipsoid is projected onto a 2D screen as an anisotropic ellipse, which contains three variables: major axis, minor axis, and direction angle. Assuming the angle is 0, we can use Figure 8 The anisotropic second image pyramid is constructed in the manner shown. Taking the second feature image with a resolution of 256*256 as an example, a 511*511 low-bit image pyramid image is constructed ( Figure 8 The last layer is not drawn in the figure. The resolution is 511 (256+128+64+32+16+8+4+2+1). In order to reduce memory and computational overhead, the second image pyramid has only one layer per octave. In the horizontal direction, the width is halved after each operation using horizontal Gaussian kernel convolution, and in the vertical direction, the width is halved after each operation using vertical Gaussian kernel convolution.

[0091] If the standard deviation of Gaussian convolution is s, S = 3*s (the influence range of Gaussian is set to 3 times the standard deviation), the length of the major and minor axes of the ellipsoid corresponding to each block is as follows: Fig. 9 As shown, the major and minor axes of the ellipsoid projected on the 2D plane can be retrieved and interpolated in the second image pyramid. Considering the angle of the ellipse, we can rotate the second feature image before constructing the second image pyramid. The preset rotation angle depends on the number of angle sampling N. N = 2 rotates 0° and 45° to construct 2 second image pyramids, N = 3 rotates 0°, 30°, and 60° to construct 3 second image pyramids, and so on. The rotation angle is a multiple of 90° / N. Taking 45° as an example, Fig.10 The long axis angle of the middle orange area (upper right) is 45°, and the long axis and short axis of the yellow area (lower left) are swapped to -45°. Similarly, the rotation preset angle of the 2D ellipse is used to obtain and interpolate values ​​in the N second image pyramids.

[0092] In step S102, a target layer is determined from each layer of the image pyramid according to the distance between the first three-dimensional coordinates of the target and the camera origin.

[0093] In some embodiments, the Gaussian radii corresponding to images at different layers in the image pyramid are different, and step S102 may include: determining the corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera; determining the difference between the Gaussian radius of each layer in the image pyramid and the target radius, and determining the two layers with the smallest difference among the layers in the image pyramid as the target layers. The camera origin is also called the camera reference point, the world coordinate origin, the camera center, etc. This point is an imaginary point that defines the position and direction of the camera in three-dimensional space. The camera origin is the basis for establishing a camera coordinate system, which is used to describe and calculate the relationship between the position of objects in the three-dimensional world and the camera.

[0094] For example, if Fig.11As shown in the figure, given the coordinates of the point X and the center of the camera C, and the focal length of the camera is f, the light is considered to be a cone, and the radius of the point X we need to retrieve is r = 0.5z / f, where z is the z-axis of X in the camera coordinates, that is, the distance from point X to point C. We use r to retrieve the two layers 11 and 12 in the Gaussian pyramid whose sigma is closest to r, which are the target layers, sigma_11≤r≤sigma_12.

[0095] In step S103, feature value retrieval is performed using the image corresponding to the target layer in the image pyramid to obtain retrieval feature values.

[0096] In some embodiments, step S103 may include: projecting the 3D cone corresponding to the target onto the image corresponding to each of the target layers, and obtaining corresponding plane triangles on the image corresponding to each layer in the target layer; performing feature value retrieval based on each of the plane triangles and the image corresponding to the target layer to obtain retrieval feature values; and calculating through trigonometric interpolation based on the retrieval feature values ​​to determine the retrieval feature values.

[0097] It is assumed that the image pyramid is as follows Figure 6 In the first image pyramid shown, the 3D cone of the target is Figure 6 The red line on the left side of the middle is projected onto the image of the target layer. After obtaining the plane triangle, we can further obtain Figure 6 The hexahedron on the right side ( Figure 6 ABCD and EFGHF represent two target layers respectively) and then determine the retrieval feature value ( Figure 6 The middle line segments IL, JK, and MN are the lines corresponding to the 3D cone of the target used in the three-dimensional difference, and O corresponds to the retrieval feature value).

[0098] In some embodiments, feature value retrieval is performed based on the images corresponding to each of the plane triangles and the target layer to obtain retrieval feature values, which may include: extracting feature values ​​of the area where the plane triangles are located in the images corresponding to each of the target layers to obtain multiple feature values ​​to be retrieved; performing matching retrieval on the multiple feature values ​​to be retrieved to obtain multiple retrieval feature values ​​that match the target.

[0099] The feature value to be retrieved is feature information related to the target in the area where the plane triangle is located in the image. A plurality of retrieval feature values ​​are obtained in the area where the plane triangle is located in the image corresponding to each target layer. The 3D cone projection of the target is determined based on the first three-dimensional coordinates of the target and the first light direction.

[0100] Through the above method, after creating the image pyramid, the feature scale to be analyzed is determined based on the distance between the first three-dimensional coordinates of the target and the camera origin, and then the target layer is selected from the image pyramid. Then, the 3D cone projection of the target is obtained by combining the first three-dimensional coordinates of the target and the first light direction, and then the plane triangle corresponding to the target on the image corresponding to each target layer is obtained, and then the feature value is retrieved based on the plane triangle and the image corresponding to the target layer to obtain the retrieval feature value. In the whole process, the first light direction is approximated as the line integral of the appropriate level in the image pyramid, and the line integral feature (that is, the retrieval feature value) is calculated through the intersection of the first light direction and the grid in the image of the target layer (that is, the plane triangle corresponding to the target), which ensures the accuracy of the feature and provides a guarantee for fast and accurate rendering.

[0101] In step S104, the search feature value is input into a preset coordinate network to predict the target volume density and target features corresponding to the target.

[0102] Among them, Fig.12 As shown, point P (x, y, z) represents the first three-dimensional coordinate of the target, which is projected onto three orthogonal feature image planes (the first feature image or the second feature image mentioned above) according to the first light direction, and each plane corresponds to brightness, chromaticity and direction information respectively. The retrieved feature values ​​are input into a small neural network, which is used to further process and adjust the features. This small neural network is usually called a regressor, and its goal is to learn how to predict the required information based on the input feature values, that is, to predict the target volume density and target features. Through training data and optimization algorithms, the regressor gradually learns how to extract useful information from the feature image and generate accurate color predictions. Finally, by solving the target volume density and target features output by the regressor, the relevant information of each pixel can be obtained.

[0103] In step S105, the target feature and the first light direction corresponding to the target are input into a preset direction network to predict the target color corresponding to the first three-dimensional coordinate.

[0104] In this embodiment, the preset coordinate network and the preset direction network may be pre-trained and used to perform corresponding information prediction to ensure the accuracy of the prediction.

[0105] In step S106, the target is rendered according to the target color and the target volume density to obtain a rendered image corresponding to the target.

[0106] like Fig.13 As shown, the embodiment of the present disclosure also provides an image processing device, the device comprising:

[0107] A pyramid creation module 71, used to create an image pyramid according to a feature image corresponding to a target, wherein the image pyramid includes a plurality of layers, each layer being an image;

[0108] a target layer determination module 72, configured to determine a target layer from each layer of the image pyramid according to a distance between the first three-dimensional coordinates of the target and a camera origin;

[0109] A feature value retrieval module 73 is used to perform feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain a retrieval feature value;

[0110] A target prediction module 74, used to input the search feature value into a preset coordinate network to predict the target volume density and target features corresponding to the target;

[0111] A color prediction module 75, configured to input the target feature and the first light direction corresponding to the target into a preset direction network, and predict the target color corresponding to the first three-dimensional coordinate;

[0112] The rendering module 76 is used to render the target according to the target color and the target volume density to obtain a rendering image corresponding to the target.

[0113] In a possible implementation manner, the Gaussian radii corresponding to images at different layers in the image pyramid are different, and the target layer determination module includes:

[0114] A radius determination submodule, used to determine a corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera;

[0115] The layer determination submodule is used to determine the difference between the Gaussian radius of each layer in the image pyramid and the target radius, and determine the two layers with the smallest difference among the layers in the image pyramid as target layers.

[0116] In a possible implementation, the feature value retrieval module includes:

[0117] A projection submodule, used for projecting the 3D cone corresponding to the target onto the image corresponding to each target layer, and obtaining corresponding plane triangles on the image corresponding to each layer in the target layer;

[0118] A retrieval submodule, used for performing feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain a retrieval feature value;

[0119] The calculation submodule is used to determine the retrieval feature value by calculating through three-property interpolation according to the retrieval feature value.

[0120] In a possible implementation, the image pyramid includes a first image pyramid, the first image pyramid includes a plurality of groups, each group includes multiple layers, and the pyramid creation module includes:

[0121] A first feature extraction submodule is used to extract features of the original image where the target is located through three planes to obtain a first feature image;

[0122] The first creation submodule is used to create a first image pyramid according to the first feature image.

[0123] In a possible implementation, the image pyramid includes a second image pyramid, the second image pyramid includes a plurality of groups, each group includes one or more layers, and the pyramid creation module includes:

[0124] A second feature extraction submodule is used to extract features from the original image where the target is located to obtain a second feature image;

[0125] An image generation submodule, used for generating an underlying image according to the second feature image;

[0126] A rotation submodule, used to rotate the bottom layer image according to a preset angle to obtain a rotated image, wherein the preset angle is determined according to the number of angle sampling;

[0127] The second creation submodule is used to repeatedly iterate according to the rotated image and the Gaussian convolution kernel size corresponding to each group to create a second image pyramid.

[0128] In a possible implementation, there is a first proportional relationship between the length of the Gaussian convolution kernel corresponding to the current group and the length of the Gaussian convolution kernel corresponding to the previous group, and there is a second proportional relationship between the width of the Gaussian convolution kernel corresponding to the current group and the width of the Gaussian convolution kernel corresponding to the previous group.

[0129] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0130] It should be noted that although the above embodiments introduce the image processing method and device as an example, those skilled in the art will understand that the present disclosure should not be limited thereto. In fact, users can flexibly set each step and module according to personal preferences and / or actual application scenarios, as long as they comply with the technical solution of the present disclosure.

[0131] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0132] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0133] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0134] Fig.14 1 is a block diagram of an apparatus 1900 for image processing according to an exemplary embodiment. For example, the apparatus 1900 may be provided as a server or a terminal device. Fig.14 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0135] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0136] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0137] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0138] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0139] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0140] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0141] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0142] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0143] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0144] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0145] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: Creating an image pyramid according to the feature image corresponding to the target, wherein the image pyramid includes a plurality of layers, each layer being an image; Determining a target layer from each layer of the image pyramid according to a distance between the first three-dimensional coordinate of the target and a camera origin; Performing feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain a retrieval feature value; Inputting the retrieved feature value into a preset coordinate network to predict the target volume density and target features corresponding to the target; Inputting the target feature and the first light direction corresponding to the target into a preset direction network to predict the target color corresponding to the first three-dimensional coordinate; Rendering the target according to the target color and the target volume density to obtain a rendered image corresponding to the target; The Gaussian radii corresponding to images at different layers in the image pyramid are different, and the target layer is determined from each layer of the image pyramid according to the distance between the first three-dimensional coordinates of the target and the camera origin, including: Determine a corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera; The difference between the Gaussian radius of each layer in the image pyramid and the target radius is determined, and two layers with the smallest difference among the layers in the image pyramid are determined as target layers.

2. The method according to claim 1, characterized in that Performing feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain retrieval feature values ​​includes: Projecting the 3D cone corresponding to the target onto the image corresponding to each target layer, and obtaining corresponding plane triangles on the image corresponding to each layer in the target layer; Perform feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain a retrieval feature value; The retrieval feature value is determined by calculating the retrieval feature value through three-dimensional interpolation.

3. The method according to claim 2, characterized in that Performing feature value retrieval based on the image corresponding to each of the plane triangles and the target layer to obtain the retrieval feature value includes: Extracting feature values ​​of the area where the plane triangle is located in the image corresponding to each target layer to obtain multiple feature values ​​to be retrieved; A matching search is performed on multiple feature values ​​to be searched to obtain multiple search feature values ​​that match the target.

4. The method according to claim 1, characterized in that: The image pyramid includes a first image pyramid, the first image pyramid includes a plurality of groups, each group includes multiple layers, and the image pyramid is created according to a feature image corresponding to a target, including: Extracting features of the original image where the target is located through three planes to obtain a first feature image; The first image pyramid is created according to the first feature image.

5. The method according to claim 1, characterized in that The image pyramid includes a second image pyramid, the second image pyramid includes a plurality of groups, each group includes one or more layers, and the image pyramid is created according to a feature image corresponding to the target, including: Extracting features from the original image where the target is located to obtain a second feature image; generating an underlying image according to the second feature image; Rotating the underlying image according to a preset angle to obtain a rotated image, wherein the preset angle is determined according to the number of angle sampling; The second image pyramid is created by repeatedly iterating according to the rotated image and the Gaussian convolution kernel size corresponding to each group.

6. The method according to claim 5, characterized in that There is a first proportional relationship between the length of the Gaussian convolution kernel corresponding to the current group and the length of the Gaussian convolution kernel corresponding to the previous group, and there is a second proportional relationship between the width of the Gaussian convolution kernel corresponding to the current group and the width of the Gaussian convolution kernel corresponding to the previous group.

7. An image processing device, characterized in that: The device comprises: A pyramid creation module, used to create an image pyramid according to a feature image corresponding to the target, wherein the image pyramid includes a plurality of layers, each layer being an image; a target layer determination module, configured to determine a target layer from each layer of the image pyramid according to a distance between the first three-dimensional coordinates of the target and a camera origin; A feature value retrieval module, used to perform feature value retrieval using the image corresponding to the target layer in the image pyramid to obtain a retrieval feature value; A target prediction module, used for inputting the retrieved feature value into a preset coordinate network to predict the target volume density and target features corresponding to the target; A color prediction module, used for inputting the target feature and the first light direction corresponding to the target into a preset direction network, and predicting the target color corresponding to the first three-dimensional coordinate; A rendering module, used for rendering the target according to the target color and the target volume density to obtain a rendering image corresponding to the target; The Gaussian radii corresponding to images at different layers in the image pyramid are different, and the target layer determination module includes: A radius determination submodule, used to determine a corresponding target radius according to the distance between the first three-dimensional coordinate and the camera origin and the focal length of the camera; The layer determination submodule is used to determine the difference between the Gaussian radius of each layer in the image pyramid and the target radius, and determine the two layers with the smallest difference among the layers in the image pyramid as target layers.

8. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 6 when executing the instructions stored in the memory.

9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images