A scene expression method and system for accurate representation of low-texture regions

Abstract

The present disclosure relates to the field of computer vision technology, and proposes a scene expression method and system for accurately representing low-texture regions, including the following steps: obtaining camera parameters, target view image coordinates, and initial depth of a camera in a scene region to be recognized; inputting the obtained data into a trained superpixel-guided scene representation network for scene expression to obtain a synthesized scene target view; the superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set to divide the entire scene into multiple local regions, and then a smooth loss function is used to optimize the scene representation of the local regions and train the superpixel-guided scene representation network. The method of the present disclosure can solve the problem of inaccurate representation of the scene expression model in low-texture regions, thereby improving the 3D structure expression ability of the scene representation model.

Description

Technical Field

[0001] The present disclosure relates to the technical field of computer vision. Specifically, it relates to a scene representation method and system for accurately representing low-texture regions. Background Art

[0002] The statements in this section merely provide background technical information related to the present disclosure and do not necessarily constitute prior art.

[0003] Neural scene representation is one of the important research directions in computer vision and is of great significance for predicting unobserved perspective image information in actual working environments. Neural scene representation implicitly reconstructs the scene, that is, represents the scene with a neural network; or, if certain perspectives cannot be obtained due to path or condition constraints, the perspective image can be generated through scene representation when the camera parameters are known.

[0004] Existing neural scene representation methods can be roughly divided into two categories: discrete scene representation methods and continuous scene representation methods. Discrete scene representation methods have a large amount of computation, and the model cannot represent all details of the scene. Different from discrete methods, continuous scene representation methods aim to learn a continuously differentiable implicit function constructed by a multi-layer perceptron (MLP). This function can realize the mapping from spatial coordinates to local scene attributes, where the local scene attributes include texture, appearance, and geometric information.

[0005] The inventors found in their research that existing continuous scene representation methods can effectively learn the appearance and geometric structure of the scene, and at the same time they also have 3D structure perception ability and can synthesize relatively more complex scene views. However, this method usually performs relatively poorly in low-texture image regions and often has the problem of local optimality, which leads to inaccurate 3D structure representation of the scene in low-texture regions. Summary of the Invention

[0006] To solve the above problems, the present disclosure proposes a scene representation method and system for accurately representing low-texture regions, which can solve the problem of inaccurate representation of the scene representation model in low-texture regions and improve the 3D structure representation ability of the scene representation model.

[0007] To achieve the above object, the present disclosure adopts the following technical solutions:

[0008] One or more embodiments provide a scene representation method for accurately representing low-texture regions, including the following steps:

[0009] Obtain the camera parameters, target view image coordinates, and initial depth of the camera in the scene region to be recognized;

[0010] Input the acquired data into the trained superpixel-guided scene representation network for scene representation to obtain a synthesized scene target view;

[0011] The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set to divide the entire scene into multiple local regions, and then a smooth loss function is used to optimize the scene representation of the local regions, calculate the regularization value as a loss function of the network, and combine it with the balanced two-dimensional image supervision loss L of the scene representation network depth and the pixel loss L Color Construct a total loss function to train the superpixel-guided scene representation network.

[0012] One or more embodiments provide a scene representation system for accurate representation of low-texture regions, including:

[0013] A data acquisition module: configured to acquire the camera parameters, the target view image coordinates (u, v), and the initial depth d0 of the camera in the scene region to be recognized;

[0014] A scene representation module: configured to input the acquired data into the trained superpixel-guided scene representation network for scene representation to obtain a synthesized scene target view;

[0015] The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set to divide the entire scene into multiple local regions, and then a smooth loss function is used to optimize the scene representation of the local regions, calculate the regularization value as a loss function of the network, and combine it with the balanced two-dimensional image supervision loss L of the scene representation network depth and the pixel loss L Color Construct a total loss function to train the superpixel-guided scene representation network.

[0016] An electronic device includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the above method are completed.

[0017] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the steps described in the above method are completed.

[0018] Compared with the prior art, the beneficial effects of the present disclosure are:

[0019] The scene expression method in the present disclosure can predict the corresponding scene according to the pose of the camera within the set area of the camera, can realize the scene synthesis of the area that cannot be directly photographed by the camera, and realizes the output of a scene view without dead angles in the entire area. The proposed superpixel regularization module can effectively alleviate the 3D structure distortion problem in low-texture areas and improve the 3D structure expression ability of the scene representation model.

[0020] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings forming a part of this disclosure are used to provide a further understanding of the present disclosure. The schematic embodiments and descriptions thereof of the present disclosure are used to explain the present disclosure and do not constitute a limitation to the present disclosure.

[0022] Figure 1 is a structural block diagram of a superpixel-guided scene representation network according to Embodiment 1 of the present disclosure;

[0023] Figure 2 is a comparison diagram of scene target images generated by using SRNs and the SG-SRNs of this embodiment respectively on different data sets in the example of Embodiment 1 of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further explanations of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present disclosure belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. It should be noted that, without conflict, the various embodiments and features in the present disclosure can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0027] Embodiment 1

[0028] In the technical solutions disclosed in one or more embodiments, as Figure 1 - Figure 2 shown, a scene expression method for accurately representing low-texture areas includes the following steps:

[0029] Step 1: Obtain the camera parameters, the target view image coordinates (u, v), and the initial depth d0 of the camera in the scene area to be recognized;

[0030] The camera parameters include the camera position and attitude data;

[0031] Step 2: Input the obtained data into the trained superpixel-guided scene representation network for scene expression to obtain the synthesized scene target view;

[0032] The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set to divide the entire scene into multiple local regions, and then a smooth loss function is used to optimize the scene representation of the local regions, calculate the regularization value as a loss function of the network, and weight it with the balanced two-dimensional image supervision loss L depth and the pixel loss L Color as the total loss function to train the superpixel-guided scene representation network;

[0033] The scene expression method in this embodiment can realize predicting the corresponding scene according to the pose of the camera within the set area of the camera, can realize the scene synthesis of the area that cannot be directly photographed by the camera, and realizes the output of the scene view without dead angles in the whole area. The proposed superpixel regularization module can effectively alleviate the 3D structure distortion problem in low-texture areas and improve the 3D structure expression ability of the scene representation model.

[0034] The method of setting the regularization module in this embodiment can be seamlessly embedded into many existing neural scene representation networks to improve the accuracy of the representation.

[0035] Optionally, the embeddable scene representation network can be SRNs (Scene Representation Networks), or NeRF (Neural Radiance Fields), or LFN (LightFieldNetworks).

[0036] SRNs is Scene Representation Networks, the scene representation network.

[0037] NeRF is Neural Radiance Fields, the neural radiance field. This network uses an MLP neural network to implicitly learn a static 3D scene.

[0038] LFN is LightFieldNetworks, the light field network.

[0039] In this embodiment, the SRNs network is adopted, and the Superpixel-Guided Scene Representation Networks (SG-SRNs) are proposed. The network structure of SG-SRNs includes two parts: the SRNs network module and the superpixel regularization module. The network structure is as Figure 1 shown.

[0040] In step 1, the camera parameters are the camera parameters of the given target view, including the camera internal parameter K and the camera pose [R t];

[0041] In step 2, the obtained camera parameters, the target view image coordinates, and the initial depth are input into the trained superpixel-guided scene representation network for scene expression to obtain the synthesized scene target view, which includes the following steps:

[0042] Step 21: The obtained camera parameters, the target view image coordinates (u, v), and the initial depth d0 data are converted into three-dimensional world coordinates x through the camera matrix;

[0043] Step 22: The converted three-dimensional world coordinates x are input into the scene representation function Φ(x) to obtain the scene representation vector v i ;

[0044] In this embodiment, the scene representation function Φ(x) is obtained by fitting through a multi-layer neural network. In the network of this embodiment, the scene representation function is composed of a fully connected neural network with 4 layers and 256 channels.

[0045] Step 23: The obtained scene representation vector v i is input into the differentiable ray marching module for iteration, and the three-dimensional world coordinates x are updated. After iteration for the set number of times n, the final scene representation vector v n is obtained;

[0046] The processing method of the differentiable ray marching module for the scene representation vector v i is specifically as follows:

[0047] The differentiable ray marching module adopts the Ray Marching Long Short-Term Memory Network (RM-LSTM). The differentiable ray marching module takes the scene representation vector v i and the previous cycle hidden state as inputs, and through network operations, a new hidden state is obtained and used as the ray marching distance δ. The depth value d i-1 of the previous cycle is added to the ray marching distance δ to obtain the updated depth value d.

[0048] Step 24: The final scene representation vector v n is input into the pixel generator for decoding to obtain the target view

[0049] Furthermore, to achieve an accurate representation of low-texture regions, a superpixel regularization module is added in the training stage in this embodiment for auxiliary training. The superpixel-guided scene representation network training method includes the following steps:

[0050] Step S1: Construct a training dataset. The dataset data includes the scene real image I, as well as the camera parameters, view image coordinates (u, v), and initial depth d0 corresponding to each target view of the scene real image I;

[0051] Among them, the camera parameters include the camera internal parameter K and the camera pose [R t];

[0052] Applying the network of this embodiment for scene representation, specifically, separate training is required for each scene, that is, after changing the scene, training for the corresponding scene is needed. The training dataset consists of images and corresponding camera parameters. The images are obtained by the camera shooting around the scene area, and the camera parameters are obtained by calibrating the camera through the checkerboard placed in the scene area.

[0053] Step S2: Obtain the camera parameters, target view image coordinates (u, v), and initial depth d0 in the training dataset, and convert them into three-dimensional world coordinates x through the camera matrix;

[0054] Step S3: Input the converted three-dimensional world coordinates x into the scene representation function Φ(x) to obtain the scene representation vector v i ;

[0055] Step S4: Input the obtained scene representation vector v i into the differentiable ray propagation module for iteration, and update the three-dimensional world coordinates x. After iterating according to the set number of times n, the final scene representation vector v is obtained n ;

[0056] Step S5: Input the final scene representation vector v n into the pixel generator for decoding to obtain the target view

[0057] Step S6: Use the scene representation vector v n , the real image I, and the superpixel mask M generated from the real image to obtain a regularization value, which is used as a loss function item to calculate the total loss function value;

[0058] The regularization value, the calculation formula is:

[0059]

[0060] Among them, (i, j) are two-dimensional image coordinates, v (i,j) is the scene representation vector at (i, j), and are the x - direction and y - direction gradients of the scene representation vector, weights and are the negative exponents of the gradients of the real image in the x and y directions. It is small in the regions with large image texture changes and large in the regions with small image texture changes, which can ensure the similarity of the attributes of adjacent spatial points. M (i,j) is the super - pixel mask. The edge pixels of its super - pixels are set to 0, and the remaining pixels are set to 1, thus eliminating the smoothing constraint on the super - pixel edges (regions with large texture changes) and retaining the smoothing constraint on the internal regions of the super - pixels (regions with small texture changes). H, W are the pixel values for traversing the image.

[0061] The calculation method of the total loss function is as follows:

[0062] 1) Using the target view and the real image I, establish the pixel loss function L Color as follows:

[0063]

[0064] where is the pixel value at the coordinate (i, j) in the generated target view, and I ij is the pixel value at the coordinate (i, j) in the real image.

[0065] 2) To make the depth d positive, establish the balanced two - dimensional image supervision loss L depth term as follows:

[0066]

[0067] where d[[ID=3)9]] ij,n represents the final depth value at the image coordinate (i, j). F]]

[0068] 3) Finally, the total loss function of SG - SRNs is: the loss function L Color and L depth of SRNs, and the weighted sum of the smoothing constraint loss function L SG-v of the regularization module, which is specifically expressed as follows:

[0069] L SG-SRNs = L Color + λ dep L depth + λ SG L SG-v (2)

[0070] where λ dep and λ SGis a hyperparameter that balances the two-dimensional image supervision loss, depth regularization, and superpixel regularizer, and can be given according to the actual value and the proportion of each loss function in the model.

[0071] Step S7: According to the calculated total loss function value, perform backpropagation, continuously iterate and update the network weights, and finally achieve the final accurate expression to obtain the trained superpixel-guided scene representation network.

[0072] The training of the SG-SRNs network in this embodiment finally uses L SG-SRNs to optimize the network parameters, so as to ensure the accuracy of the texture-rich area while reducing the problem of inaccurate expression in the low-texture area.

[0073] To illustrate the effect of the method in this embodiment, a comparative experiment was carried out with the original SRNs method on the Synthetic-NeRF dataset and the Block dataset, and the depth estimation index was used to quantify the experimental data results. The depth estimation indexes include absolute relative difference (AbsRel), root mean square error (RMSE), RMSE(log), relative error (SqRel), and three threshold precisions (δ < 1.05, δ < 1.1, and δ < 1.15). The smaller the first four indexes and the larger the last three indexes, the higher the depth estimation accuracy. The comparison of the depth estimation indexes is shown in Table 1. Except that the results of the two are similar in the Synthetic-NeRF (mic) dataset, the SG-SRNs in this embodiment obtains better results than the original SRN method, that is, the three-dimensional structure representation accuracy of the SG-SRNs is higher.

[0074] Table 1

[0075]

[0076] The comparison effect of the generated target image is as Figure 2 shown. In the comparison of the depth maps, the original SRNs differ greatly from the real sparse depth map. For example, in the ground area of the Block-Real image, the depth predicted by the SRNs is larger than the real depth, resulting in serious three-dimensional structure distortion. In contrast, the SG-SRNs can better handle the planar area. In addition, the edge between the object and the background represented by the original SRNs is relatively blurred, and the contour is wrongly enlarged. For example, in the ear area of the bunny model in the Synthetic-NeRF (bunny) dataset. The SG-SRNs performs relatively well in edge representation. In summary, the SG-SRNs can represent the three-dimensional structure of the scene more accurately than the original SRNs.

[0077] Embodiment 2

[0078] Based on Embodiment 1, a scene expression system for accurately representing low-texture regions is provided in this embodiment, including:

[0079] Data acquisition module: configured to acquire the camera parameters, the target view image coordinates (u, v), and the initial depth d0 of the camera in the scene region to be recognized;

[0080] Scene expression module: configured to input the acquired data into the trained superpixel-guided scene representation network for scene expression to obtain a synthesized scene target view;

[0081] The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set to divide the entire scene into multiple local regions, and then a smooth loss function is used to optimize the scene representation of the local regions, calculate the regularization value as a loss function of the network, and balance the two-dimensional image supervision loss L of the scene representation network depth and the pixel loss L Color Construct a total loss function to train the superpixel-guided scene representation network.

[0082] It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1, and the specific implementation process is the same, so it will not be repeated here.

[0083] Embodiment 3

[0084] This embodiment provides an electronic device, including a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of Embodiment 1 are completed.

[0085] Embodiment 4

[0086] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by the processor, the steps described in the method of Embodiment 1 are completed.

[0087] The above are only the preferred embodiments of the present disclosure and are not used to limit the present disclosure. For those skilled in the art, the present disclosure can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

[0088] Although the specific implementation manners of the present disclosure are described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present disclosure. Those skilled in the art should understand that based on the technical solutions of the present disclosure, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of the present disclosure.

Claims

1. A scene expression method for accurate representation of low-texture regions, characterized in that It includes the following steps: Obtain the camera parameters, target view image coordinates, and initial depth of the camera in the scene area to be recognized; Input the obtained data into the trained superpixel-guided scene representation network for scene representation to obtain the synthesized scene target view; The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set up to divide the entire scene into multiple local regions. Then, a smooth loss function is used to optimize the scene representation of the local regions, and the regularization value is calculated as a loss function of the network, which is combined with the balanced two-dimensional image supervision loss of the scene representation network and the pixel loss A total loss function is constructed to train the superpixel-guided scene representation network; The training method of the superpixel-guided scene representation network includes the following steps: Construct a training data set, and the data in the data set includes the real scene image, as well as the camera parameters, view image coordinates, and initial depth corresponding to each target view of the real scene image; Obtain the camera parameters in the training dataset, as well as the target view image coordinates and the initial depth, and convert them into three-dimensional world coordinates through the camera matrix ; Input the converted three-dimensional world coordinates into the scene representation function to obtain the scene representation vector ; The obtained scene representation vector is input into the differentiable ray marching module for iteration, and the three-dimensional world coordinates are updated . After iteration for the set number of times n, the final scene representation vector is obtained ; The differentiable ray marching module uses a ray marching long short-term memory network. With the scene representation vector and the previous cycle hidden state as inputs, a new hidden state is obtained through network operations and used as the ray marching distance δ. Adding the depth value di-1 of the previous cycle to the ray marching distance δ gives the updated depth value d Final scene representation vector Input to the pixel generator for decoding to obtain the target view ; Using a scene representation vector , a real image and a superpixel mask generated from the real image , a regularization value is obtained as a loss function item to calculate the total loss function value; According to the calculated total loss function value, perform backpropagation, continuously iterate and update the network weights, and finally achieve the final accurate representation to obtain the trained superpixel-guided scene representation network.

2. The method for scene expression for precise representation of low-texture regions according to claim 1, characterized in that: The camera parameters include camera position and attitude data.

3. A scene expression method for accurately representing low-texture regions as claimed in claim 1, characterized in that: The scene representation network is an SRNs network, a NeRF network, or an LFN network.

4. A scene expression method for accurate representation of low-texture regions as described in claim 1, characterized in that: The scene representation function is specifically obtained by fitting through a multi-layer neural network.

5. A method for scene expression for precise representation of low-texture regions according to claim 1, characterized in that: The camera parameters for a given target view include camera intrinsics and camera attitude; Alternatively, the total loss function value is the balanced two-dimensional image supervision loss , the pixel loss and the smoothing constraint loss function of the superpixel regularization module weighted sum.

6. A scene expression system for precise representation of low-texture regions, characterized in that, It includes: Data acquisition module: configured to obtain the camera parameters, target view image coordinates, and initial depth of the camera in the scene area to be recognized; Scene representation module: configured to input the obtained data into the trained superpixel-guided scene representation network for scene representation to obtain the synthesized scene target view; The superpixel-guided scene representation network includes a scene representation network and a superpixel regularization module. Before the pixel generator of the scene representation network, a superpixel regularization module is set up to divide the entire scene into multiple local regions. Then, using a smooth loss function, the scene representation of the local regions is optimized, and the regularization value is calculated as a loss function of the network, which is combined with the balanced two-dimensional image supervision loss of the scene representation network and the pixel loss Construct a total loss function to train the superpixel-guided scene representation network; The training method of the superpixel-guided scene representation network includes the following steps: Construct a training data set, and the data in the data set includes the real scene image, as well as the camera parameters, view image coordinates, and initial depth corresponding to each target view of the real scene image; Obtain the camera parameters in the training dataset, as well as the target view image coordinates and initial depth, and convert them into three-dimensional world coordinates through the camera matrix ; Input the converted three-dimensional world coordinates , into the scene representation function to obtain the scene representation vector ; The obtained scene representation vector is input into the differentiable ray marching module for iteration, and the three-dimensional world coordinates are updated . After iteration for the set number of times n, the final scene representation vector is obtained ; The differentiable ray marching module uses a ray marching long short-term memory network. With the scene representation vector and the previous cycle's hidden state as inputs, a new hidden state is obtained through network operations and used as the ray marching distance δ. Adding the previous cycle's depth value di-1 to the ray marching distance δ gives the updated depth value d Final scene representation vector Input to the pixel generator for decoding to obtain the target view ; Using a scene representation vector , a real image and a superpixel mask generated from the real image , a regularization value is obtained as a loss function item to calculate the total loss function value; According to the calculated total loss function value, perform backpropagation, continuously iterate and update the network weights, and finally achieve the final accurate representation to obtain the trained superpixel-guided scene representation network.

7. An electronic device, characterized in that, It includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are run by the processor, the steps of the method according to any one of claims 1-5 are completed.

8. A computer-readable storage medium, characterized in that, For storing computer instructions, when the computer instructions are executed by the processor, the steps of the method according to any one of claims 1-5 are completed.

Images