View image processing method, device, electronic device and computer storage medium

By constructing a distribution-driven process based on attention mechanism, capturing the hidden scene distribution and calculating the occlusion weight, the problems of poor background texture recovery and unsatisfactory judging of occlusion relationships in the existing technology are solved, and high-quality new perspective synthesis of dynamic scenes is achieved.

CN116228855BActive Publication Date: 2024-11-15BEIJING PHIGENT TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211737720.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-30
Publication Date
2024-11-15
Estimated Expiration
2042-12-30

AI Technical Summary

Technical Problem

The existing new perspective synthesis method for dynamic scenes is poor when dealing with scenes containing motion foreground and static backgrounds, and the background texture recovery effect is poor, and the object occlusion relationship is only judged through RGB space, which is not ideal.

Method used

By determining the camera pose parameters, a distribution-driven process is constructed based on the attention mechanism, the hidden scene distribution is captured, and the occlusion weight of the ray sampling points is calculated, and high-quality new perspective images are generated based on the rendering results of the background and dynamic neural radiation field.

Benefits of technology

Effectively learn the occlusion relationship between the foreground and the background, improve the effect of synthesis of new perspectives in dynamic scenes, generate high-quality images, and provide a 3D paradigm for separating the foreground and the background.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116228855B_ABST
    Figure CN116228855B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for processing a perspective image, which relates to the field of image processing. The method includes: determining the camera pose parameters of the original perspective image; determining the hidden scene distribution based on the camera pose parameters, and sampling from the hidden scene distribution to obtain distribution sampling points; determining the first volume density and the first color rendering result of the background neural radiation field based on the camera pose parameters and the distribution sampling points; determining the second volume density and the second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step; determining the first transmission weight of each light in the original perspective image based on the camera pose parameters; rendering the first volume density, the first color rendering result, the second volume density and the second color rendering result based on the first transmission weight to obtain a first target perspective image. The embodiment of the present invention generates a high-quality new perspective synthetic image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for processing a perspective image, a device for processing a perspective image, an electronic device and a computer-readable storage medium. Background Art

[0002] New perspective synthesis in dynamic scenes refers to learning to restore any new viewpoint image that does not appear in the video when given a video clip captured by one or more cameras. It can generate unlimited perspective images under the condition of limited view input and has a wide range of applications. For example, it can create special effects of switching perspectives in movies and games, render images in AR (Augmented Reality) and VR (Virtual Reality) worlds, or assist camera imaging to achieve robot and autonomous driving perception and navigation interaction.

[0003] In recent years, new perspective synthesis methods for dynamic scenes have been improved based on neural radiance fields. The radiance field method based on semantic segmentation network can supervise the learning of neural radiance field with semantic information, but it is limited by the performance of the semantic segmentation network itself and is therefore only applicable to the same specific fields as the segmentation network. The radiance field method based on temporal information uses one or more neural radiance fields to learn dynamic information in video sequences, but only focuses on the features between adjacent frames and cannot capture the continuous information of the entire scene.

[0004] In existing methods, for scenes containing a single object, such as indoor Lego scenes, the switching of perspectives will cause a huge change in the 2D scene projection on the camera plane. Unlike these scenes, dynamic scenes in the real world usually contain moving foregrounds and static backgrounds. These static backgrounds (such as blocks, buildings, etc.) are far away from the camera lens. Existing methods only focus on the temporal information between video frame sequences, resulting in poor texture recovery effect on the background.

[0005] In addition, existing methods only judge the occlusion relationship between objects by fusing the rendering results in the RGB space, which shows unsatisfactory performance in judging the occlusion relationship between static background and dynamic objects. Summary of the invention

[0006] In view of the above problems, embodiments of the present invention are proposed to provide a method for processing a perspective image, a device for processing a perspective image, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0007] In order to solve the above problem, an embodiment of the present invention discloses a method for processing a viewing angle image, the method comprising:

[0008] Determine the camera pose parameters of the original view image;

[0009] Determine hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points;

[0010] Determine a first volume density and a first color rendering result of a background neural radiation field based on the camera pose parameters and the distributed sampling points;

[0011] Determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step;

[0012] Determine a first transmission weight of each light ray in the original perspective image based on the camera pose parameter;

[0013] The first volume density, the first color rendering result, the second volume density, and the second color rendering result are rendered based on the first transmission weight to obtain a first target viewing angle image.

[0014] In one or more embodiments, determining the hidden scene distribution based on the camera pose parameters, and sampling from the hidden scene distribution to obtain distribution sampling points includes:

[0015] The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution;

[0016] Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

[0017] In one or more embodiments, the camera pose parameters include a three-dimensional position and a three-dimensional angle;

[0018] The determining a first volume density and a first color rendering result of a background neural radiation field based on the camera pose parameters and the distributed sampling points includes:

[0019] Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron;

[0020] Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value;

[0021] The first attention value and the three-dimensional position are added to obtain a first volume density of the background neural radiation field, and the second attention value and the three-dimensional angle are added to obtain a first color rendering result of the background neural radiation field.

[0022] In one or more embodiments, the camera pose parameters and time step are also used to determine the optical flow of the dynamic neural radiance field;

[0023] The method further comprises:

[0024] The dynamic neural radiation field is updated using the optical flow to obtain an updated dynamic neural radiation field.

[0025] In one or more embodiments, determining a first transmission weight of each light ray in the original perspective image based on the camera pose parameter comprises:

[0026] Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters;

[0027] A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

[0028] In one or more embodiments, further comprising:

[0029] In response to the camera pose adjustment instruction, determining a parameter offset based on the adjusted camera pose parameters and the camera pose parameters;

[0030] Determining the adjusted second transmission weight of each light ray, the third volume density and the third color rendering result of the background neural radiation field, and the fourth volume density and the fourth color rendering result of the dynamic neural radiation field based on the parameter offset;

[0031] The third volume density, the third color rendering result, the fourth volume density, and the fourth color rendering result are rendered based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

[0032] Correspondingly, an embodiment of the present invention further discloses a viewing angle image processing device, the device comprising:

[0033] A first determination module is used to determine the camera pose parameters of the original view image;

[0034] A first processing module, configured to determine a hidden scene distribution based on the camera pose parameters, and to sample from the hidden scene distribution to obtain distribution sampling points;

[0035] A second processing module, configured to determine a first volume density and a first color rendering result of a background neural radiation field based on the camera posture parameters and the distributed sampling points;

[0036] A third processing module, configured to determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step;

[0037] A fourth processing module, configured to determine a first transmission weight of each light ray in the original perspective image based on the camera posture parameter;

[0038] A rendering module is used to render the first volume density, the first color rendering result, the second volume density and the second color rendering result based on the first transmission weight to obtain a first target viewing angle image.

[0039] In one or more embodiments, the first processing module is specifically configured to:

[0040] The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution;

[0041] Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

[0042] In one or more embodiments, the camera pose parameters include a three-dimensional position and a three-dimensional angle;

[0043] The second processing module is specifically used for:

[0044] Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron;

[0045] Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value;

[0046] The first attention value and the three-dimensional position are added to obtain a first volume density of the background neural radiation field, and the second attention value and the three-dimensional angle are added to obtain a first color rendering result of the background neural radiation field.

[0047] In one or more embodiments, the camera pose parameters and time step are also used to determine the optical flow of the dynamic neural radiance field;

[0048] The device also includes:

[0049] An updating module is used to update the dynamic neural radiation field using the optical flow to obtain an updated dynamic neural radiation field.

[0050] In one or more embodiments, the fourth processing module is specifically configured to:

[0051] Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters;

[0052] A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

[0053] In one or more embodiments, further comprising:

[0054] A second determination module is used to determine a parameter offset based on the adjusted camera pose parameters and the camera pose parameters in response to the camera pose adjustment instruction;

[0055] The second processing module is also used for the third volume density and third color rendering results of the background neural radiation field;

[0056] The third processing module is further used for a fourth volume density and a fourth color rendering result of the dynamic neural radiation field;

[0057] The fourth processing module is further used to determine the adjusted second transmission weight of each light ray based on the parameter offset;

[0058] The rendering module is further used to render the third volume density, the third color rendering result, the fourth volume density and the fourth color rendering result based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

[0059] Correspondingly, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the various steps of the above-mentioned perspective image processing method embodiment when executed by the processor.

[0060] Accordingly, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned perspective image processing method embodiment are implemented.

[0061] The embodiments of the present invention include the following advantages:

[0062] Determine the camera pose parameters of the original perspective image, determine the hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points, then determine the first volume density and the first color rendering result of the background neural radiation field based on the camera pose parameters and the distribution sampling points, determine the second volume density and the second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step, and determine the first transmission weight of each light in the original perspective image based on the camera pose parameters, and then render the first volume density, the first color rendering result, the second volume density and the second color rendering result based on the first transmission weight to obtain a first target perspective image. The embodiment of the present invention constructs a distribution-driven process based on the attention mechanism, so that the hidden scene distribution in the overall scene can be captured. At the same time, a calculation method for the occlusion weight of the light sampling point in the neural radiation field is also constructed, and the occlusion relationship between the foreground object and the background is effectively learned, so as to add an occlusion weight to each light, thereby improving the effect of synthesizing a new perspective in a dynamic scene, thereby generating a high-quality new perspective synthetic image, and providing a 3D paradigm for separating the foreground and the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flowchart of the steps of Embodiment 1 of a method for processing a viewing angle image of the present invention;

[0064] Figure 2 This is the overall process of the attention mechanism of the present invention;

[0065] Figure 3 is the structure of the transmission weight network of the present invention;

[0066] Figure 4 It is a logical flow diagram of the present invention;

[0067] Figure 5 is a flowchart of the steps of Embodiment 2 of a method for processing a viewing angle image of the present invention;

[0068] Figures 6A-6B are the original viewing angle image and the second target viewing angle image of the present invention;

[0069] Figure 7 It is a structural block diagram of an embodiment of a viewing angle image processing device of the present invention. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] One of the core concepts of the embodiment of the present invention is to determine the camera pose parameters of the original perspective image, determine the hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points, and then determine the first volume density and the first color rendering result of the background neural radiation field based on the camera pose parameters and the distribution sampling points, determine the second volume density and the second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step, and determine the first transmission weight of each light in the original perspective image based on the camera pose parameters, and then render the first volume density, the first color rendering result, the second volume density and the second color rendering result based on the first transmission weight to obtain a first target perspective image. The embodiment of the present invention constructs a distribution-driven process based on the attention mechanism, so that the hidden scene distribution in the overall scene can be captured. At the same time, a calculation method for the occlusion weight of the light sampling point in the neural radiation field is also constructed, and the occlusion relationship between the foreground object and the background is effectively learned, so as to add an occlusion weight to each light, improve the effect of synthesizing a new perspective in a dynamic scene, and then generate a high-quality new perspective synthetic image, and provide a 3D paradigm for separating the foreground and the background.

[0072] Reference Figure 1 , shows a flowchart of a first embodiment of a method for processing a viewing angle image of the present invention, which may specifically include the following steps:

[0073] Step 101, determining the camera pose parameters of the original view image.

[0074] The original view image may be an image captured by the camera at the original view, and the image may be a picture or a video containing video frames. Since in the real world, the camera has real world coordinates when shooting, the image can be calculated to determine the pose parameters of the camera when shooting the image. The pose parameters of the camera include but are not limited to the three-dimensional position (denoted as "emb(x)") and three-dimensional angle (denoted as "emb(d)") of the camera.

[0075] Furthermore, the camera pose parameters may be determined by calculation, by taking pictures with a depth camera, or by other methods. In practical applications, they may be adjusted according to actual needs, and the embodiments of the present invention do not limit this.

[0076] Step 102: determine the hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points.

[0077] After determining the camera pose parameters, the hidden scene distribution of the entire scene in the image can be determined according to the camera pose parameters, and then sampling is performed from the hidden scene distribution to obtain distribution sampling points.

[0078] In an embodiment of the present invention, determining the hidden scene distribution based on the camera pose parameters, and sampling from the hidden scene distribution to obtain distribution sampling points includes:

[0079] The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution;

[0080] Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

[0081] Specifically, a distribution encoder can be deployed in advance. During sampling, the distribution encoder uses a multi-layer perceptron (MLP) to learn the three-dimensional position and three-dimensional angle to obtain the hidden scene distribution (denoted as "p(z|θ)"), and then samples from the hidden scene distribution to obtain a distribution sampling point (denoted as "z").

[0082] Step 103: Determine a first volume density and a first color rendering result of the background neural radiation field based on the camera pose parameters and the distributed sampling points.

[0083] After obtaining the distribution sampling points, the distribution sampling points can be combined with the background neural radiation field according to the camera pose parameters, and the volume density of the background neural radiation field can be calculated using the attention mechanism (denoted as the first volume density, σ b ) and color rendering result (recorded as the first color rendering result, C b ).

[0084] In an embodiment of the present invention, the determining a first volume density and a first color rendering result of a background neural radiation field based on the camera pose parameters and the distributed sampling points includes:

[0085] Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron;

[0086] Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value;

[0087] The first attention value and the three-dimensional position are added to obtain a first volume density of the background neural radiation field, and the second attention value and the three-dimensional angle are added to obtain a first color rendering result of the background neural radiation field.

[0088] Figure 2 The overall process of the attention mechanism is shown. Specifically, the distribution sampling points and the three-dimensional position can be scaled to the same attention space (referred to as the first attention space) through a multi-layer perceptron (MLP). Under this dimension, the distribution sampling points are used as key values ​​(K, V) and the three-dimensional position is used as a query (Q) to calculate the attention, thereby obtaining the attention value (referred to as the first attention value). Then, the first attention value and the three-dimensional position information are added to obtain the volume density of the background neural radiation field (referred to as the first volume density).

[0089] Similarly, the distributed sampling points and three-dimensional angles can be scaled to the same attention space (referred to as the second attention space) through a multi-layer perceptron (MLP). At this dimensional size, the distributed sampling points are used as key values ​​(K, V) and the three-dimensional angles are used as queries (Q) to calculate the attention, thereby obtaining the attention value (referred to as the second attention value). The second attention value and the three-dimensional angle information are then added together to obtain the color rendering result of the background neural radiation field (referred to as the first color rendering result).

[0090] Step 104: determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step.

[0091] For the dynamic neural radiation field, a network that considers timing information can be constructed. The input of the network includes three-dimensional position, three-dimensional angle and time step. The input of the network includes the volume density of the dynamic neural radiation field (denoted as the second volume density), the color rendering result (denoted as the second color rendering result), and the optical flow of the dynamic neural radiation field (denoted as F () ).

[0092] In an embodiment of the present invention, the method further includes:

[0093] The dynamic neural radiation field is updated using the optical flow to obtain an updated dynamic neural radiation field.

[0094] Specifically, optical flow can be used to supervise the modeling of the dynamic part of the dynamic neural radiation field learning between different frames. That is, the dynamic neural radiation field can be updated (back propagation) using optical flow to obtain an updated dynamic neural radiation field.

[0095] Step 105: Determine a first transmission weight of each light ray in the original perspective image based on the camera pose parameters.

[0096] In the real world, when observing from a certain perspective, the foreground will block the background, and the light in the blocked area is different from the light in the unblocked area. Therefore, in order to show the occlusion relationship of light in the image, the transmission weight of each light in the original perspective image can be calculated according to the camera posture parameters (recorded as the first transmission weight).

[0097] In an embodiment of the present invention, determining a first transmission weight of each light ray in the original perspective image based on the camera pose parameter includes:

[0098] Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters;

[0099] A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

[0100] Figure 3 The structure of the transmission weight network is shown, where H and W represent the length and width of the image, x and y are the coordinates of the camera, and N p is the number of light sampling points, and k represents the kth one. That is to say, for any light in the image, the occlusion weight (denoted as "Ω") of each light sampling point in the light can be calculated through the network, and then the transmission weight of the light can be calculated according to the occlusion weight of each light sampling point. Similarly, the transmission weight of each light can be calculated.

[0101] Step 106: Render the first volume density, the first color rendering result, the second volume density, and the second color rendering result based on the first transmission weight to obtain a first target viewing angle image.

[0102] After obtaining the first transmission weight, the first volume density, the first color rendering result, the second volume density and the second color rendering result, the first transmission weight can be used to mix and render the first volume density, the first color rendering result, the second volume density and the second color rendering result to obtain a new perspective image (recorded as the first target perspective image).

[0103] For ease of understanding, Figure 4The logical flow of an embodiment of the present invention is shown. Specifically, after determining the three-dimensional position and three-dimensional angle of the camera, the three-dimensional position and the three-dimensional angle are input into a distribution encoder to obtain a hidden scene distribution, and then sampling is performed therefrom to obtain distribution sampling points, and then the distribution sampling points, the three-dimensional position, and the three-dimensional angle are input into the background neural radiation field to obtain a first volume density and a first color rendering result. At the same time, the three-dimensional position and the three-dimensional angle are input into the occlusion weight and the transmission weight network to obtain a first transmission weight. Further, the three-dimensional position and the three-dimensional angle are input into the dynamic neural radiation field to obtain a second volume density, a second color rendering result, and an optical flow. Based on the first transmission weight, the first volume density, the first color rendering result, the second volume density, and the second color rendering result are mixed to obtain a first target perspective image.

[0104] In an embodiment of the present invention, the camera pose parameters of the original perspective image are determined, the hidden scene distribution is determined based on the camera pose parameters, and sampling is performed from the hidden scene distribution to obtain distribution sampling points, and then the first volume density and the first color rendering result of the background neural radiation field are determined based on the camera pose parameters and the distribution sampling points, the second volume density and the second color rendering result of the dynamic neural radiation field are determined based on the camera pose parameters and the time step, and the first transmission weight of each light in the original perspective image is determined based on the camera pose parameters, and then the first volume density, the first color rendering result, the second volume density and the second color rendering result are rendered based on the first transmission weight to obtain a first target perspective image. The embodiment of the present invention constructs a distribution-driven process based on the attention mechanism, so that the hidden scene distribution in the overall scene can be captured. At the same time, a calculation method for the occlusion weight of the light sampling point in the neural radiation field is also constructed, and the occlusion relationship between the foreground object and the background is effectively learned, so as to add an occlusion weight to each light, thereby improving the effect of synthesizing a new perspective in a dynamic scene, thereby generating a high-quality new perspective synthetic image, and providing a 3D paradigm for separating the foreground and the background.

[0105] Reference Figure 5 , shows a flowchart of a second embodiment of a method for processing a viewing angle image of the present invention, which may specifically include the following steps:

[0106] Step 501, determining the camera pose parameters of the original perspective image.

[0107] Step 502: determine the hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points.

[0108] Step 503: Determine a first volume density and a first color rendering result of the background neural radiation field based on the camera pose parameters and the distributed sampling points.

[0109] Step 504: determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step.

[0110] Step 505: Determine a first transmission weight of each light ray in the original perspective image based on the camera pose parameters.

[0111] Step 506: Render the first volume density, the first color rendering result, the second volume density, and the second color rendering result based on the first transmission weight to obtain a first target viewing angle image.

[0112] Among them, steps 501 to 506 are substantially the same as steps 101 to 106, and will not be described here for avoiding repetition.

[0113] Step 507 , in response to the camera pose adjustment instruction, determining a parameter offset based on the adjusted camera pose parameters and the camera pose parameters.

[0114] Step 508, determining the adjusted second transmission weight of each light ray, the third volume density and third color rendering result of the background neural radiation field, and the fourth volume density and fourth color rendering result of the dynamic neural radiation field based on the parameter offset.

[0115] Step 509: Render the third volume density, the third color rendering result, the fourth volume density, and the fourth color rendering result based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

[0116] Specifically, the user can adjust the camera's posture to change the viewing angle. When the user adjusts the camera's posture to the target posture, the offset of the posture parameters can be calculated based on the adjusted posture and the original posture. Then, the transmission weight (recorded as the second transmission weight), the volume density of the background neural radiation field (recorded as the third volume density), the color rendering result (recorded as the third color rendering result), the volume density of the dynamic neural radiation field (recorded as the fourth volume density) and the color rendering result (recorded as the fourth color rendering result) of each ray corresponding to the target posture are calculated based on the offset. The third volume density, the third color rendering result, the fourth volume density and the fourth color rendering result are mixed and rendered using the second transmission weight to obtain the camera's viewing angle image at the target posture (recorded as the second target viewing angle image).

[0117] Fig. 6A The original view image is shown. After adjusting the camera’s position, we can get Figure 6B The second target perspective image is shown.

[0118] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0119] Reference Figure 7 , shows a structural block diagram of an embodiment of a viewing angle image processing device of the present invention, which may specifically include the following modules:

[0120] A first determination module 701 is used to determine the camera pose parameters of the original perspective image;

[0121] A first processing module 702 is used to determine the hidden scene distribution based on the camera pose parameters, and to sample from the hidden scene distribution to obtain distribution sampling points;

[0122] A second processing module 703 is used to determine a first volume density and a first color rendering result of a background neural radiation field based on the camera posture parameters and the distribution sampling points;

[0123] A third processing module 704 is used to determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step;

[0124] A fourth processing module 705 is used to determine a first transmission weight of each light in the original perspective image based on the camera posture parameter;

[0125] The rendering module 706 is configured to render the first volume density, the first color rendering result, the second volume density, and the second color rendering result based on the first transmission weight to obtain a first target viewing angle image.

[0126] In the embodiment of the present invention, the first processing module is specifically used to:

[0127] The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution;

[0128] Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

[0129] In an embodiment of the present invention, the camera pose parameters include a three-dimensional position and a three-dimensional angle;

[0130] The second processing module is specifically used for:

[0131] Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron;

[0132] Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value;

[0133] The first attention value and the three-dimensional position are added to obtain a first volume density of the background neural radiation field, and the second attention value and the three-dimensional angle are added to obtain a first color rendering result of the background neural radiation field.

[0134] In an embodiment of the present invention, the camera pose parameters and time step are also used to determine the optical flow of the dynamic neural radiation field;

[0135] The device also includes:

[0136] An updating module is used to update the dynamic neural radiation field using the optical flow to obtain an updated dynamic neural radiation field.

[0137] In the embodiment of the present invention, the fourth processing module is specifically used to:

[0138] Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters;

[0139] A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

[0140] In an embodiment of the present invention, it also includes:

[0141] A second determination module is used to determine a parameter offset based on the adjusted camera pose parameters and the camera pose parameters in response to the camera pose adjustment instruction;

[0142] The second processing module is also used for the third volume density and third color rendering results of the background neural radiation field;

[0143] The third processing module is further used for a fourth volume density and a fourth color rendering result of the dynamic neural radiation field;

[0144] The fourth processing module is further used to determine the adjusted second transmission weight of each light ray based on the parameter offset;

[0145] The rendering module is further used to render the third volume density, the third color rendering result, the fourth volume density and the fourth color rendering result based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

[0146] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0147] An embodiment of the present invention further provides an electronic device, including:

[0148] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned perspective image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0149] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned perspective image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0150] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0151] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0152] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0156] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0157] The above is a detailed introduction to a method for processing a perspective image and a device for processing a perspective image provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for processing a perspective image, It is characterized in that The method comprises: Determine the camera pose parameters of the original view image; Determine the hidden scene distribution based on the camera pose parameters, and sample from the hidden scene distribution to obtain distribution sampling points; the camera pose parameters include a three-dimensional position and a three-dimensional angle; Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron; Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value; Adding the first attention value and the three-dimensional position to obtain a first volume density of a background neural radiation field, and adding the second attention value and the three-dimensional angle to obtain a first color rendering result of the background neural radiation field; Determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step; Determine a first transmission weight of each light ray in the original perspective image based on the camera pose parameter; The first volume density, the first color rendering result, the second volume density, and the second color rendering result are rendered based on the first transmission weight to obtain a first target viewing angle image.

2. The method for processing a perspective image according to claim 1, It is characterized in that The determining of the hidden scene distribution based on the camera pose parameters and sampling from the hidden scene distribution to obtain distribution sampling points includes: The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution; Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

3. The method for processing a perspective image according to claim 1, It is characterized in that The camera pose parameters and time step are also used to determine the optical flow of the dynamic neural radiance field; The method further comprises: The dynamic neural radiation field is updated using the optical flow to obtain an updated dynamic neural radiation field.

4. The method for processing a perspective image according to claim 1, It is characterized in that The determining a first transmission weight of each light ray in the original perspective image based on the camera pose parameter comprises: Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters; A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

5. The method for processing a perspective image according to claim 1, It is characterized in that Also includes: In response to the camera pose adjustment instruction, determining a parameter offset based on the adjusted camera pose parameters and the camera pose parameters; Determining the adjusted second transmission weight of each light ray, the third volume density and the third color rendering result of the background neural radiation field, and the fourth volume density and the fourth color rendering result of the dynamic neural radiation field based on the parameter offset; The third volume density, the third color rendering result, the fourth volume density, and the fourth color rendering result are rendered based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

6. A viewing angle image processing device, It is characterized in that The device comprises: A first determination module is used to determine the camera pose parameters of the original view image; A first processing module, configured to determine a hidden scene distribution based on the camera pose parameters, and to sample from the hidden scene distribution to obtain distribution sampling points; the camera pose parameters include a three-dimensional position and a three-dimensional angle; The second processing module is specifically used for: Scaling the distribution sampling points and the three-dimensional positions to a first attention space through a multi-layer perceptron, and scaling the distribution sampling points and the three-dimensional angles to a second attention space through a multi-layer perceptron; Calculating the attention value of the first attention space using the distribution sampling point as a key value and the three-dimensional position as a query to obtain a first attention value, and calculating the attention value of the second attention space using the distribution sampling point as a key value and the three-dimensional angle as a query to obtain a second attention value; Adding the first attention value and the three-dimensional position to obtain a first volume density of a background neural radiation field, and adding the second attention value and the three-dimensional angle to obtain a first color rendering result of the background neural radiation field; A third processing module, configured to determine a second volume density and a second color rendering result of the dynamic neural radiation field based on the camera pose parameters and the time step; A fourth processing module, configured to determine a first transmission weight of each light ray in the original perspective image based on the camera posture parameter; A rendering module is used to render the first volume density, the first color rendering result, the second volume density and the second color rendering result based on the first transmission weight to obtain a first target viewing angle image.

7. The viewing angle image processing device according to claim 6, It is characterized in that The first processing module is specifically used for: The camera pose parameters are learned by using a multi-layer perceptron through a distribution encoder to obtain a hidden scene distribution; Sampling is performed from the hidden scene distribution to obtain distribution sampling points.

8. The viewing angle image processing device according to claim 6, It is characterized in that The camera pose parameters and time step are also used to determine the optical flow of the dynamic neural radiance field; The device also includes: An updating module is used to update the dynamic neural radiation field using the optical flow to obtain an updated dynamic neural radiation field.

9. The viewing angle image processing device according to claim 6, It is characterized in that The fourth processing module is specifically used for: Calculate the occlusion weight of each sampling point of each light ray in the original perspective image based on the camera pose parameters; A first transmission weight of each ray is calculated based on the occlusion weight of each sampling point of each ray.

10. The viewing angle image processing device according to claim 6, It is characterized in that Also includes: A second determination module is used to determine a parameter offset based on the adjusted camera pose parameters and the camera pose parameters in response to the camera pose adjustment instruction; The second processing module is also used for the third volume density and third color rendering results of the background neural radiation field; The third processing module is further used for a fourth volume density and a fourth color rendering result of the dynamic neural radiation field; The fourth processing module is further used to determine the adjusted second transmission weight of each light ray based on the parameter offset; The rendering module is further used to render the third volume density, the third color rendering result, the fourth volume density and the fourth color rendering result based on the second transmission weight to obtain a second target perspective image after the camera posture is adjusted.

11. An electronic device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the method for processing a perspective image as claimed in any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for processing a perspective image according to any one of claims 1 to 5 are implemented.