Change detection method based on nerve radiation field

Abstract

The invention discloses a change detection method based on a neural radiation field, and the method comprises the steps: carrying out the rendering through employing a trained scene NeRF model before change according to a default rendering path of a corresponding scene, generating a rendered image before change, and generating a rendered image after change through employing the same rendering path and employing the changed scene NeRF model; calculating the density and weight of each sampling point by using an MLP network in the trained NeRF model, and taking a point with the maximum weight value and four adjacent points near the point as change points; traversing all pixels of the difference image to obtain a change area point cloud corresponding to the difference image; de-noising operation is performed on the point cloud based on the view angle consistency difference to obtain a change point cloud; when the image is rendered under the new view angle, the change pixels are marked by judging whether the light passes through the point cloud or not, and finally a rendered image with a change mark scene is generated.

Description

Technical Field

[0001] The present invention relates to the field of change detection, and in particular to a change detection method based on neural radiation fields. Background Art

[0002] The purpose of change detection is to identify the changed areas or objects in the images taken before and after the scene changes. This has many applications in remote sensing, monitoring, and robot navigation. Change detection can be divided into two categories: one is two-dimensional change detection, which has been widely used in remote sensing and street scene scenes. The basic requirement is that there should be no obvious difference in the image observation angle before and after the change. The other is three-dimensional change detection, which can reflect the changes in three-dimensional space through multi-view before and after images and three-dimensional scanning.

[0003] In 2D change detection, the image change (difference) task aims to locate the changed object and describe the change in natural language. Its development history is rich and diverse. As summarized in the literature [1], IDC proposed to use fine-grained features, namely visual features, semantic features and instance-level position features, as the representation of objects to accurately locate the changed objects; VACC proposed a viewpoint-independent change captioning network with cycle-consistent differences, thereby improving the robustness of change detection captured with a large camera perspective; MCCFormers proposed a multi-change captioning transformer to identify the change region by densely correlating different regions in the image pair and dynamically determine the change region associated with the words in the sentence; HPCFNet designed an effective feature fusion method to improve the accuracy of the corresponding change map; Dr-tanet proposed temporal attention and explored the impact of the change range size of image attention on change detection performance as time changes. CYWS aims to detect "object-level" changes in image pairs under different viewpoints. CYWS-3D further studies change detection with significant changes in camera pose. Although two-dimensional change detection technology is already very powerful and comprehensive, it requires image alignment and training with a large number of change image pairs. It can only generate change maps at the viewpoint of the input image, and it is difficult to obtain change maps at different viewpoints.

[0004] Three-dimensional change detection can reflect the changes in three-dimensional space through multi-view before and after images and three-dimensional scans, thus avoiding the problems of two-dimensional change detection. Previous methods have used dense color and depth information to detect changes between three-dimensional maps. There are also three-dimensional reconstruction algorithms based on extended truncated signed distance functions, which can more robustly solve the scene difference problem. Recent methods propose to explicitly locate the changes in the three-dimensional bounding box from two pairs of point clouds before and after the change, and describe the detailed scene changes. However, the above-mentioned previous three-dimensional change detection methods are only suitable for change detection in large-scale scenes such as street scenes and cities, and cannot perform fine-grained change detection. At the same time, these methods are also very limited in their ability to express scenes.

[0005] References

[0006] [1] Yang Bin, Mao Yin, Chen Jin, Liu Jianqiang, Chen Jie, Yan Kai. A review of remote sensing change detection based on deep learning: bibliometrics and analysis[J]. Journal of Remote Sensing, 2023, 27(9): 1988-2005

[0007] [2]Johannes L Schonberger and Jan-Michael Frahm.Structurefrom-motionrevisited.In Proceedings of the IEEE conerence on computer vision and patternrecognition,pages 4104-4113,2016.

[0008] [3] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neuralradiance fields for view synthesis. In ECCV, 2020.

[0009] [4]Kajiya,JT,Herzen,BPVRay tracing volumedensities.Comput.Graph.(SIGGRAPH)(1984).

[0010] [5] Park, JJ, Florence, P., Straub, J., Newcombe, R., Lovegrove, S. DeepSDF: Learning continuous signed distance functions for shape representation. InCVPR (2019). Summary of the invention

[0011] The present invention provides a change detection method based on neural radiation field. The present invention aligns the images before and after the change by the SfM method to obtain the scenes before and after the change in the same coordinate system; then the scenes before and after the change are trained respectively to generate two NeRF models before and after the change, and the images of the scenes before and after the change aligned at the pixel level are rendered using a given virtual camera posture; the absolute difference method is used to obtain the difference map; based on the difference map, the change light is located, the weights of the sampling points on the light are calculated to obtain the change point cloud corresponding to each difference map, and based on the difference in perspective consistency, the change point cloud is denoised by using corresponding strategies for the two scenes of direct view and surround; the image with the change mark is rendered at a new perspective, as described below:

[0012] A change detection method based on neural radiation field, the method comprising:

[0013] Using the trained NeRF model of the scene before the change, rendering is performed according to the default rendering path of the corresponding scene to generate the rendered image before the change, and using the same rendering path and the NeRF model of the scene after the change to generate the rendered image after the change;

[0014] The density and weight of each sampling point are calculated using the MLP network in the trained NeRF model, and the point with the largest weight value and its four adjacent points are taken as the change point; all pixels in the difference map are traversed to obtain the change area point cloud corresponding to the difference map;

[0015] The point cloud is denoised based on the difference in perspective consistency to obtain a point cloud of changed points; when rendering the image under a new perspective, the changed pixels are marked by judging whether the light passes through the point cloud, and finally a rendered image with a changed marked scene is generated.

[0016] The point cloud obtained by performing denoising operation on the point cloud based on the difference in perspective consistency is:

[0017] For the direct view scene, the point clouds corresponding to each difference map are intersected one by one to obtain the denoised point cloud:

[0018]

[0019] Among them, the symbol ∩ is defined as the point cloud intersection operation, then:

[0020]

[0021] For the surround scene, take the corresponding point clouds of the viewpoints at m equally divided points on the N viewpoints of the circular rendering path; merge the m point clouds to get a set of point clouds All point clouds are grouped to obtain M groups of point clouds, and the intersection of the M groups of point clouds is taken to obtain the surrounding scene denoised point cloud P. The grouping and intersection expressions are:

[0022]

[0023]

[0024] Among them, f(·,r,λ) is the radius filter, r is the filter radius, and λ is the number of points.

[0025] When rendering an image under a new perspective, marking changed pixels by judging whether the light passes through the point cloud is specifically as follows:

[0026] The denoised change point cloud P is replaced one by one with voxels of side length h, and each ray emitted from the camera origin through the pixel center is detected. If any sample point of the ray is located in the voxel grid, the color of the ray is changed to a given value. The calculation method of the ray color is:

[0027]

[0028] Among them, V represents the set of changing voxels, r∩V means that the sampling point on the ray is located in the voxel set V; when When cv1=(0,0,0) and cv2=(1,1,1), the rendering result is a binary image.

[0029] For surround scenes, the following formula is used to determine whether the light color is blocked:

[0030]

[0031] Among them, d v Indicates the voxel on the ray that is closest to the camera origin, d w Indicates the shortest distance from the camera origin to the object.

[0032] The beneficial effects of the technical solution provided by the present invention are:

[0033] 1. The present invention uses the SfM method to register the images before and after the scene changes, obtain the camera pose when the image is taken, and make the NeRF model before and after the change in the same coordinate system;

[0034] 2. Based on the principle of perspective consistency difference, the present invention adopts different strategies to denoise the change point cloud for direct view scenes and surround scenes, which can effectively remove the noise caused by rendering quality and background differences;

[0035] 3. The present invention only needs to perform the change detection and denoising process once. After obtaining the coordinates of the change point, the change detection and denoising process does not need to be performed again in the subsequent new perspective rendering process;

[0036] 4. After obtaining the coordinates of the change point, the present invention can obtain a new perspective rendering image with a change mark given any valid virtual camera posture, thereby realizing change detection under different perspectives;

[0037] 5. The present invention designs a change detection method based on neural radiation field, combines the absolute difference method to locate the change point in space, and denoises based on the difference in perspective consistency, so that the detection results are more in line with actual detection needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the flow of the change detection method based on neural radiation field proposed by the present invention;

[0039] Figure 2 This is a schematic diagram of the image capture path proposed by the present invention;

[0040] Figure (a) is a schematic diagram of the shooting path for a direct-view scene, and Figure (b) is a schematic diagram of the shooting path for a surround scene;

[0041] Figure 3 A schematic diagram of the neural radiation field rendering and training process used in the present invention;

[0042] Figure 4 A schematic diagram of the neural radiation field MLP network structure used in the present invention;

[0043] Figure 5 A schematic diagram of a surround scene denoising method proposed by the present invention;

[0044] Figure 6 This is a result diagram of the example change detection proposed by the present invention;

[0045] Figure 7 A comparison chart of the fine-grained change detection proposed in the present invention. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0047] The embodiment of the present invention proposes a change detection method based on neural radiation fields to address the problem of change detection in three-dimensional scenes. The method can generate a rendered image of the scene with change marks at a specified viewing angle, so that the image can more comprehensively reflect the change information of the scene.

[0048] Example 1

[0049] A change detection method based on neural radiation fields, see Figure 1 , the method comprises the following steps:

[0050] 1. NeRF of the scene before and after the change [3] Model Generation

[0051] The specific steps are as follows: According to the scene requirements, use Figure 3 The two shooting paths (a) and (b) in the figure are used to shoot images before and after the scene changes. Then, the two sets of images are placed in the same folder in the order before and after the change. Then, colmap is used [2] To calculate the image pose, the pose order is consistent with the image order. After obtaining the pose corresponding to the image, the image and pose before and after the change are placed in two different folders for training the NeRF model. During the training process, the corresponding pose can be read according to actual needs.

[0052] In the embodiment of the present invention, the training process of the NeRF model is as follows: Figure 3 As shown. The position of the camera at the time of shooting can be determined based on the pose corresponding to the image. The light emitted from the origin of the camera passes through the pixels, and these rays are simulated by the sampling points; given the three-dimensional coordinates (x, y, z) and directions θ, φ of the sampling points, position encoding is used to encode the position and direction information to a higher dimension to retain the high-frequency detail information of the image, and then the encoded position and direction information are sent to the MLP network respectively, and the MLP network returns the color c of the sampling point i and density σ i Usually, 64 or 128 sampling points are sampled for a ray. After obtaining the color and density of all sampling points of a ray, the voxel rendering formula is used. [4] The discrete form of is:

[0053]

[0054] After calculation, the color value of the light is obtained, that is, the predicted pixel value of the pixel corresponding to the light, and the L2 loss function is calculated with the corresponding real pixel value of the captured image to optimize the network parameters.

[0055] in, α i =1-exp(-σ i δi ),δ i =t i+1 -t i is the distance between two adjacent sampling points.

[0056] Among them, the MLP network structure is as follows Figure 4 As shown in Figure 1, it contains 10 hidden layers, all of which are fully connected networks. The positional encoding of the input will pass through 8 fully connected ReLU layers, each with 256 channels. [5] The architecture uses a jump connection to connect the input to the 5th hidden layer. The 8th hidden layer outputs a density and a 256-dimensional feature vector, which is connected to the position encoding of the input viewing direction, and then processed by a fully connected ReLU layer with 128 channels, and finally the sigmoid activation function is used to obtain the color of the sampling point.

[0057] The embodiment of the present invention follows the paradigm of the native NeRF model and adopts two sets of identical MLP networks. The network structure is as follows: Figure 4 As shown in the figure, a hierarchical sampling strategy from coarse to fine is adopted for each ray. Specifically, the first set of MLP networks, here referred to as the coarse network, follows the above sampling and training strategy, but stops after the density of the sampling points is output at the 8th layer. After that, according to the density distribution function constructed by the density value generated by the coarse network, a number of sampling points (64 or 128) are sampled again for each ray in the high-density part, that is, the part that is more likely to be the surface of the object. Then, the initial uniform sampling points and the re-sampled fine-grained sampling points are combined into a sampling point set and sent to the second set of MLP networks, that is, the fine network for complete training. In the stage of rendering a new view, only the fine network is used for reasoning.

[0058] Based on the above process, the Pytorch deep learning framework is used to build and train two sets of NeRF models before and after the change for use in subsequent processes.

[0059] 2. Coarse Change Detection

[0060] The embodiment of the present invention uses the trained NeRF model before the scene changes, and renders according to the default rendering path of the corresponding scene, thereby generating a rendered image before the change. Next, the same rendering path is also used to generate a rendered image after the change using the NeRF model after the change. The two sets of images correspond one to one, preparing for subsequent analysis.

[0061] In the difference analysis stage, the embodiment of the present invention uses the absolute difference method to process the rendered image pair. The absolute difference method is a technology used in image processing and computer vision, and is mainly used to compare the difference between two images. Its basic idea is to identify changes or differences by calculating the absolute difference of corresponding pixel values ​​in two images. The embodiment of the present invention uses the algorithm to calculate the difference between the two images before and after the rendering change, and generates a difference map. In this process, the threshold of the absolute difference method can be adjusted according to the characteristics of the specific scene to ensure the accuracy and operability of the analysis. In the difference map finally generated, the pixel values ​​of the white area are (255,255,255), indicating that there are significant changes in these areas; and the pixel values ​​of the black area are (0,0,0), indicating that these areas have not changed. In this way, the embodiment of the present invention can effectively identify the changed areas in the scene, which is convenient for subsequent analysis and processing.

[0062] 3. False Change Filtering

[0063] Traverse the previous step to get the difference map. For each difference map, locate the changed pixels and determine the changed light corresponding to the changed pixels. Then, use the MLP network in the trained NeRF model to calculate the density σ of each sampling point. i and weight w i =T i α i , take the point with the largest weight value and its four adjacent points as the change point; traverse all pixels of the difference map and execute this process to obtain the change area point cloud P corresponding to the difference map i .

[0064] The difference in perspective consistency is specifically manifested as follows: in the difference graphs at different perspectives, the changing area always appears stably, but the noise is unstable. The same area may have noise at one perspective but no noise at another perspective, but the changing area will not change significantly with the change of perspective.

[0065] Based on this difference in perspective consistency, for the direct view scene, the point clouds corresponding to each difference map are intersected one by one in turn to obtain the denoised point cloud:

[0066]

[0067] The symbol ∩ is defined as the point cloud intersection operation, for example: P in =P1∩P2, P1 and P2 are point clouds, then:

[0068]

[0069] That is, if the Euclidean distance between two points is less than d, then the two points are considered to intersect.

[0070] For surround scenes, such as Figure 5 As shown in FIG. 1 , it is necessary to obtain the corresponding point clouds of the viewpoints at m equally divided points on the N viewpoints of the circular rendering path; then, these m point clouds are merged to obtain a set of point clouds Grouping all point clouds in this way can obtain M groups of point clouds. Taking the intersection of the M groups of point clouds, we can obtain the surrounding scene denoised point cloud P. The grouping and intersection expressions are as follows:

[0071]

[0072]

[0073] Among them, f(·,r,λ) is the radius filter, r is the filter radius, and λ is the number of points.

[0074] 4. New Perspective Change Detection

[0075] Replace the denoised change point cloud P with voxels of side length h one by one, detect each ray from the camera origin through the pixel center, and change the color of the ray to a given value if any sample point of the ray is in the voxel grid. The ray color is calculated as:

[0076]

[0077] Among them, V represents the set of changing voxels, r∩V means that the sampling point on the ray is located in the voxel set V; when When cv1=(0,0,0) and cv2=(1,1,1), the rendering result is a binary image.

[0078] For surround scenes, voxels may be blocked by other objects in the scene at a certain viewing angle. At this time, although the light passes through the voxel, the color of the light passing through the blocked voxel should not be changed because the voxel is blocked. The following formula is used to determine whether the light color is blocked:

[0079]

[0080] Among them, d v Indicates the voxel on the ray that is closest to the camera origin, d w Indicates the shortest distance from the camera origin to the object.

[0081] In summary, the embodiments of the present invention use colmap to jointly calculate the shooting poses of the images before and after the change, so as to train the two NeRF models before and after the change, so that they are in the same coordinate system; given the same virtual camera pose, the NeRF models before and after the change can render aligned image pairs, and obtain the difference map based on the absolute difference method, and on the basis of the difference map, obtain the point cloud corresponding to each difference map, and then perform denoising operation on these point clouds based on the difference in perspective consistency, so as to obtain the point cloud of the changed points; when rendering the image under the new perspective, the changed pixels are marked by judging whether the light passes through the point cloud, and this method meets various needs in practical applications.

[0082] Example 2

[0083] Combine the following Figure 6 , Figure 7 The feasibility of the solution in Example 1 is verified as follows:

[0084] according to Figure 2 The image capture path shown in the figure prepares images before and after the scene change, and uses colmap to calculate the pose of the images before and after the change. Figure 2 , Figure 3 The network structure and training process shown in the figure build and train the NeRF model before and after the scene change in the embodiment of the present invention. Given the same pose, use two trained NeRF models to render aligned images, and obtain the difference map of the rendered image pair based on the absolute difference method; map the difference area in the difference map to a point cloud, and denoise the point cloud of the change point based on the difference in perspective consistency. If it is a direct view scene, take the intersection of the point clouds before and after the change; if it is a surround scene, use the method based on Figure 5 The process shown performs surround scene denoising; then a new view image with change markers is rendered at any given view.

[0085] from Figure 6 It can be seen that the embodiment of the present invention can accurately use boxes to mark the changed objects in the scene in the instance change detection. Compared with the existing method of performing change detection based on two images with different viewing angles, the present method can accurately mark the changed objects and can output results in multiple viewing angles to better present the change information of the scene.

[0086] from Figure 7 It can be seen that the embodiments of the present invention can perform effective fine-grained change detection. Figure 7Two groups of change detection results are shown in Figure 1. In each group, the first two rows are aligned image pairs rendered by the NeRF model before and after the change, the third row is the difference map of the first two rows of images, and the fourth row is the change detection result, where the arrows in the first row of images point to the changed area in the scene. Taking the first group as an example, the first, third, and fourth columns perform change detection on the two-dimensional scene. It can be seen from the results that the embodiments of the present invention can effectively detect subtle changes in the scene.

[0087] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A change detection method based on neural radiation field, characterized in that: The method comprises: Using the trained NeRF model of the scene before the change, rendering is performed according to the default rendering path of the corresponding scene to generate the rendered image before the change, and using the same rendering path and the NeRF model of the scene after the change to generate the rendered image after the change; The density and weight of each sampling point are calculated using the MLP network in the trained NeRF model, and the point with the largest weight value and its four adjacent points are taken as the change point; all pixels in the difference map are traversed to obtain the change area point cloud corresponding to the difference map; The point cloud is denoised based on the difference in perspective consistency to obtain a point cloud of changed points; when rendering the image under a new perspective, the changed pixels are marked by judging whether the light passes through the point cloud, and finally a rendered image with a changed marked scene is generated.

2. The change detection method based on neural radiation field according to claim 1, characterized in that: The point cloud obtained by performing denoising operation on the point cloud based on the difference in perspective consistency is: For the direct view scene, the point clouds corresponding to each difference map are intersected one by one to obtain the denoised point cloud: Among them, the symbol ∩ is defined as the point cloud intersection operation, then: For the surround scene, take the corresponding point clouds of the viewpoints at m equally divided points on the N viewpoints of the circular rendering path; merge the m point clouds to get a set of point clouds All point clouds are grouped to obtain M groups of point clouds, and the intersection of the M groups of point clouds is taken to obtain the surrounding scene denoised point cloud P. The grouping and intersection expressions are: Among them, f(·,r,λ) is the radius filter, r is the filter radius, and λ is the number of points.

3. The change detection method based on neural radiation field according to claim 1 is characterized in that: When rendering an image under a new perspective, marking the changed pixels by judging whether the light passes through the point cloud is specifically as follows: The denoised change point cloud P is replaced one by one with voxels of side length h, and each ray emitted from the camera origin through the pixel center is detected. If any sample point of the ray is located in the voxel grid, the color of the ray is changed to a given value. The calculation method of the ray color is: Among them, V represents the set of changing voxels, r∩V means that the sampling point on the ray is located in the voxel set V; when When cv1=(0,0,0) and cv2=(1,1,1), the rendering result is a binary image. For surround scenes, the following formula is used to determine whether the light color is blocked: Among them, d v Indicates the voxel on the ray that is closest to the camera origin, d w Indicates the shortest distance from the camera origin to the object.

Images