Image change segmentation method based on view angle alignment

Abstract

The invention relates to an image change segmentation method based on view angle alignment, and belongs to the technical field of image recognition. The method comprises the following steps: acquiring an original image pair, and preprocessing the images to obtain two preprocessed images; the two preprocessed images pass through a 2D scene alignment module to obtain aligned images corresponding to the two images; inputting the two preprocessed images and the aligned image into a feature extraction module for feature extraction to obtain feature information corresponding to the two images; inputting the feature information into a preliminary change detection network to obtain difference information corresponding to the two images; the difference information is input into a feature fusion module for feature fusion, and change information corresponding to the two images is obtained; and inputting the change information into a positioning network for frame detection to obtain bounding boxes of respective change areas of the two images. According to the method, the image registration difference structure and the fusion structure are constructed and combined, so that the image change can be accurately detected.

Description

Technical Field

[0001] The invention belongs to the technical field of image recognition, and in particular relates to an image change segmentation method based on view angle alignment. Background Art

[0002] Being able to accurately detect and localize changes in both 3D and 2D scenes remains a challenging task in computer vision. Imagine being able to identify changes between two images of the same scene taken from different viewpoints at different times. This is the challenge addressed in this work. With applications in areas such as robotics, facility monitoring, forensics, and augmented reality, our work has the potential to open new possibilities for understanding and interacting with a dynamic world. We formulate the problem as follows: Given a pair of RGB images of any scene (3D or 2D), captured with significant changes in camera position and over time, we wish to localize the changes between them. In particular, we wish to capture everything that is physically different in the regions visible in the two images, while ignoring regions that appear or disappear from view due to changes in camera pose or occlusion. This includes objects that may have been added or removed from the scene, as well as text or decorations that may have been added to objects, while ignoring photometric differences, such as lighting changes. Furthermore, in order to operate only on RGB images without taking into account additional information such as camera parameters and camera pose, we are able to detect changes between pairs of images regardless of the scene. Therefore, how to provide reliable and accurate change detection regardless of the influence of various scenarios and environments is a challenging problem. Summary of the invention

[0003] In order to overcome the deficiencies of the above technologies, the present invention provides an image change segmentation method based on view angle alignment.

[0004] In order to achieve the above object, the present invention is implemented through the following technical solutions: In a first aspect, the present invention provides an image change segmentation method based on perspective alignment, comprising the following steps: S1. Obtain an original image pair, and preprocess the image pair to obtain two preprocessed images; S2. The two preprocessed images are passed through a 2D scene alignment module to obtain an aligned image corresponding to the two images; S3. Construct a feature extraction module, input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction, and obtain feature information corresponding to the two images; S4. constructing a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and inputting feature information corresponding to the two aligned images into the preliminary change detection network to obtain difference information corresponding to the two images; S5. Construct a feature fusion module, input the difference information corresponding to the two images into the feature fusion module for feature fusion, and obtain the change information corresponding to the two images; S6. Construct a positioning network for border detection, input the change information corresponding to the two images into the positioning network for border detection, and obtain the bounding boxes of the changed areas of the two images respectively.

[0005] Furthermore, step S1 is specifically as follows: Perform geometric transformation on the two original images and obtain the preprocessed size as Left image and right image , , Indicates that the data type of the matrix elements is real number, is the height of the image, is the width of the image, 3 is the number of channels of the image, Representing images It is a shape with a size of The real matrix of Same reason.

[0006] Furthermore, the 2D scene alignment module in step S2 is composed of an image feature point matching unit and a homography registration unit, specifically: S21. Preprocessed left image and right image The image feature point matching unit uses the feature matching extractor Extract feature points and get the left image Matching feature point set and right image Matching feature point set , while the preprocessed left image and right image Using a monocular depth estimator Get the corresponding depth map respectively ; S22. The feature point set and After the homography registration unit calculates the homography transformation matrix, we get The transformation matrix ,and The transformation matrix , the transformation matrix and Apply to the corresponding image to achieve image alignment and get the left image Aligned images in 2D scenes based on the right image R and right image Aligned images in 2D scenes based on the left image L , .

[0007] Furthermore, the feature extraction module in step S3 includes a baseline feature extraction model and a U-Net encoder, specifically: S31. The preprocessed left image , right image after preprocessing , left image Aligned images of and right image Aligned images of Input into the benchmark feature extraction model for feature extraction to obtain the corresponding preliminary feature information , the preprocessed left image Corresponding preliminary feature information With the right image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the left image L , the preprocessed right image Corresponding preliminary feature information With the left image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the right image R ; S32. and Input them into their respective U-Net compilers and output them respectively and Corresponding five intermediate feature maps of different scales and , When s=1, the scale of the feature map is 64×64, when s=2, the scale of the feature map is 32×32, when s=3, the scale of the feature map is 16×16, when s=4, the scale of the feature map is 8×8, and when s=5, the scale of the feature map is 4×4.

[0008] Furthermore, the preliminary change detection network in step S4 includes a 3D scene image registration module and a difference module, and the difference module is composed of a subtraction operation, specifically: S41. The corresponding intermediate feature map and The corresponding intermediate feature map Perform the channel half-division operation respectively to obtain the left image The corresponding feature map ,image The corresponding feature map , right image The corresponding feature map and images The corresponding feature map ,in, ; S42. In the 3D scene image registration module, the preprocessed left image is obtained according to step S21. The corresponding feature point set , right image The corresponding feature point set And the preprocessed left image The corresponding depth map And the right image The corresponding depth map , first for each point in the feature point set Perform back projection to get the projection vector , and then get the feature point set in homogeneous coordinates The corresponding sparse 3D point clouds , according to the sparse 3D point cloud through the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , and according to the sparse three-dimensional point cloud, the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , the formula is as follows: , , , in, express The depth value of express The matrix generalized inverse of express The matrix generalized inverse of ; S43. The left image The corresponding feature map The image coordinates are used to obtain the 3D linear transformation matrix in step S42 Transform it into Image coordinates in a three-dimensional coordinate system , use the point cloud tool to convert the converted image coordinates and feature map Encapsulate into a point cloud object, and finally render the point cloud object into a feature map through a differentiable renderer Aligning image features based on 3D scene correspondence , ; Similarly, we get the feature map Aligning image features based on 3D scene correspondence , ; S44. The left image The corresponding feature map ,image The corresponding feature map And feature map Aligning image features based on 3D scene correspondence Input into the difference module, and through the subtraction operation, the left image is obtained Based on the difference information in 2D scene and left image Based on the difference information in 3D scene , the formula is as follows: , Similarly, the right image R is obtained based on the difference information in the 2D scene And the right image R based on the difference information in the 3D scene .

[0009] Furthermore, the feature fusion module in step S5 is specifically: The difference information and Perform splicing to obtain splicing features, and input the splicing features into the linear layer to obtain intermediate features , and then perform convolution operation to get the weight , the weight With the left image The corresponding feature map Multiply element by element and finally get the left image Change information , the formula is as follows: , , , in, Indicates that two features are merged into one channel. represents a nonlinear activation function, represents the first convolutional layer, represents the second convolutional layer, represents the Sigmoid activation function, Represents an element-by-element multiplication operation; similarly, we get the right image Change information .

[0010] Further, step S6 is specifically as follows: The change information and After upsampling and decoding by U-Net decoder, the left image is obtained The corresponding feature map and right image The corresponding feature map , and then the feature map and The data is input into the CenterNet network head for border detection and positioning to obtain the bounding boxes of the changed areas of the two images.

[0011] In a second aspect, an image change segmentation device based on perspective alignment includes: Data acquisition unit: used to acquire the original image pair and preprocess the image pair to obtain two preprocessed images; 2D scene alignment unit: used for passing the two pre-processed images through the 2D scene alignment module to obtain an aligned image corresponding to the two images; Feature extraction unit: used to input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction to obtain feature information corresponding to the two images; A first construction unit is used to construct a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and feature information corresponding to two aligned images is input into the preliminary change detection network to obtain difference information corresponding to the two images; The second construction unit is used to construct a feature fusion module, and input the difference information corresponding to the two images into the feature fusion module for feature fusion to obtain the change information corresponding to the two images; The third construction unit is used to construct a positioning network for border detection, and input the change information corresponding to the two images into the positioning network for border detection to obtain the boundary boxes of the change areas of the two images respectively.

[0012] According to a third aspect, an electronic device comprises: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image change segmentation method based on perspective alignment as described in the first aspect are performed.

[0013] In a fourth aspect, a computer-readable storage medium is provided, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the image change segmentation method based on perspective alignment as described in the first aspect are executed.

[0014] The advantages of the present invention are: The present invention adopts a network composed of an image registration difference structure and a fusion structure. The RGB image pairs are aligned respectively by two image registration methods to obtain two types of difference information, one of which is biased towards 2D scenes and the other is biased towards 3D scenes. Then, the two types of difference information are effectively fused through the fusion structure to make up for the problem of insufficient change difference information. At the same time, the network adopts a Siamese neural network architecture, so that two images can be operated at the same time to better complete the recognition of the change area. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0016] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a 3D-based image registration operation diagram of the present invention; Figure 3 It is an operation diagram of the difference module and the feature fusion module of the present invention; Figure 4 A network structure diagram of the image change segmentation method based on view alignment of the present invention; Figure 5 This is the segmentation result diagram of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1 In this embodiment, Figure 1 As shown, the present invention provides an image change segmentation method based on perspective alignment, and the specific steps include: S1. Obtain an original image pair, and preprocess the image pair to obtain two preprocessed images; Specifically, the two original images are geometrically transformed to obtain the preprocessed size of Left image and right image , , Indicates that the data type of the matrix elements is real number, is the height of the image, is the width of the image, 3 is the number of channels of the image, Representing images It is a shape with a size of The real matrix of Same reason.

[0019] S2. The two preprocessed images are passed through a 2D scene alignment module to obtain an aligned image corresponding to the two images; Specifically, the 2D scene alignment module is composed of an image feature point matching unit and a homography registration unit; Specifically, S21. The preprocessed left image and right image The image feature point matching unit uses the feature matching extractor Extract feature points and get the left image Matching feature point set and right image Matching feature point set , while the preprocessed left image and right image Using a monocular depth estimator Get the corresponding depth map respectively .

[0020] Specifically, S22. The feature point set and After the homography registration unit calculates the homography transformation matrix, we get The transformation matrix ,and The transformation matrix , the transformation matrix and Apply to the corresponding image to achieve image alignment and get the left image Aligned images in 2D scenes based on the right image R and right image Aligned images in 2D scenes based on the left image L , .

[0021] S3. Construct a feature extraction module, input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction, and obtain feature information corresponding to the two images; Specifically, the feature extraction module includes a baseline feature extraction model and a U-Net encoder; Specifically, S31. The preprocessed left image , right image after preprocessing , left image Aligned images of and right image Aligned images of Input into the benchmark feature extraction model for feature extraction to obtain the corresponding preliminary feature information , the preprocessed left image Corresponding preliminary feature information With the right image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the left image L , the preprocessed right image Corresponding preliminary feature information With the left image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the right image R ; Specifically, S32. and Input them into their respective U-Net compilers and output them respectively and Corresponding five intermediate feature maps of different scales and , When s=1, the scale of the feature map is 64×64, when s=2, the scale of the feature map is 32×32, when s=3, the scale of the feature map is 16×16, when s=4, the scale of the feature map is 8×8, and when s=5, the scale of the feature map is 4×4.

[0022] S4. constructing a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and inputting feature information corresponding to the two aligned images into the preliminary change detection network to obtain difference information corresponding to the two images; Specifically, S41. The corresponding intermediate feature map and The corresponding intermediate feature map Perform the channel half-division operation respectively to obtain the left image The corresponding feature map ,image The corresponding feature map , right image The corresponding feature map and images The corresponding feature map ,in, ; Specifically, in the 3D scene image registration module, the preprocessed left image is obtained according to step S21. The corresponding feature point set , right image The corresponding feature point set And the preprocessed left image The corresponding depth map And the right image The corresponding depth map , first for each point in the feature point set Perform back projection to get the projection vector , and then get the feature point set in homogeneous coordinates The corresponding sparse 3D point clouds , according to the sparse 3D point cloud through the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , and according to the sparse three-dimensional point cloud, the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , the formula is as follows: , , , in, express The depth value of express The matrix generalized inverse of express The matrix generalized inverse of .

[0023] Specifically, S43. The left image The corresponding feature map The image coordinates are used to obtain the 3D linear transformation matrix in step S42 Transform it into Image coordinates in a three-dimensional coordinate system , use the point cloud tool to convert the converted image coordinates and feature map Encapsulate into a point cloud object, and finally render the point cloud object into a feature map through a differentiable renderer Aligning image features based on 3D scene correspondence , ; Similarly, we get the feature map Aligning image features based on 3D scene correspondence , .

[0024] Specifically, S44. The left image The corresponding feature map ,image The corresponding feature map And feature map Aligning image features based on 3D scene correspondence Input into the difference module, and through the subtraction operation, the left image is obtained Based on the difference information in 2D scene and left image Based on the difference information in 3D scene , the formula is as follows: , Similarly, the right image R is obtained based on the difference information in the 2D scene And the right image R based on the difference information in the 3D scene .

[0025] S5. Construct a feature fusion module, input the difference information corresponding to the two images into the feature fusion module for feature fusion, and obtain the change information corresponding to the two images; Specifically, Figure 3 As shown, in the feature fusion module, the difference information and Perform splicing to obtain splicing features, and input the splicing features into the linear layer to obtain intermediate features , and then perform convolution operation to get the weight , the weight With the left image The corresponding feature map Multiply element by element and finally get the left image Change information , the formula is as follows: , , , in, Indicates that two features are merged into one channel. represents a nonlinear activation function, represents the first convolutional layer, represents the second convolutional layer, represents the Sigmoid activation function, Represents an element-by-element multiplication operation; similarly, we get the right image Change information .

[0026] S6. Construct a positioning network for border detection, input the change information corresponding to the two images into the positioning network for border detection, and obtain the bounding boxes of the changed areas of the two images respectively.

[0027] Specifically, the change information and After upsampling and decoding by U-Net decoder, the left image is obtained The corresponding feature map and right image The corresponding feature map , and then the feature map and The data is input into the CenterNet network head for border detection and positioning to obtain the bounding boxes of the changed areas of the two images.

[0028] Example 2 In this embodiment, in order to verify the effectiveness of the method of the present invention, the KC-3D dataset, RC-3D dataset, COCO-Inpainted dataset, Synthtext-Change dataset, VIRAT-STD dataset, and Kubric-Change dataset are used for evaluation. The COCO-Inpainted dataset is a test set based on changes that we have compiled from the COCO test subset. We divide the test set into three categories according to the size of the changing object, namely small, medium, and large. All represents the integration of the test sets of the three categories. We have compiled 1655 pairs of images for small objects, 1747 pairs of images for medium objects, and 1006 pairs of images for large objects, totaling 4408 pairs of images for the COCO-Inpainted test set.

[0029] The KC-3D dataset uses the Kubric dataset generator to curate 86,407 pairs of 3D scene images with controlled variations (4,548 of which are used for validation). The scene consists of randomly selected 3D objects generated at random positions on a random texture plane, which we iteratively remove and capture "before" and "after" image pairs. We capture images of various camera poses in a cylindrical space around the object. An additional 4,548 pairs of 3D scene images were extracted from Kubric for testing.

[0030] The RC-3D dataset quantifies the model's ability to generalize to real-world images. A small-scale test set of 100 pairs of images was manually collected and labeled, capturing a variety of common objects found in everyday places such as offices, kitchens, lounges, etc. These images were taken with a handheld Apple iPad Pro (4th generation) and Apple iPhone 14 Pro, which have built-in lidars that provide aligned RGB-D images.

[0031] The Synthtext-Change dataset adds random text to the "background" image through synthesis techniques and generates 5000 pairs of images in a way that is consistent with its geometry. To detect changes in outdoor scenes, we randomly select 1000 pairs of images from the STD dataset. Since STD does not provide basic GroundTruth for changes, an automated tool is used to obtain the basic GroundTruth. Since the camera is static, there is a same geometric transformation between images, but the photometric conditions may change due to time of day, weather conditions, etc. The Kubric-Change dataset is 1605 pairs of realistic images of changes. The scene consists of a set of randomly selected 3D objects that are located on a randomly textured ground plane. For a given scene, objects are iteratively removed from it and "before" and "after" image pairs are captured.

[0032] For quantitative evaluation, we calculate the average precision (AP) as the evaluation metric based on the predicted bounding box and the ground-truth bounding box according to previous related methods.

[0033] Figure 4 The network structure diagram of the method of the present invention is shown in Table 1. The performance comparison between the classic image change detection algorithm and the method of the present invention is shown in Table 1. The experiment is set for 12 epochs, and the optimization method Adam is used. The default learning rate is 0.00001, and the weight decay is 0.0005. In order to enhance the model's ability to fit the data, we have adopted random affine transformation, contrast enhancement, illumination enhancement and saturation enhancement. Table 1 Comparison of the performance of the currently optimal change detection model and the method of the present invention on different data sets: The CYWS-3D model is the best change detection model in the current research field. From the table, we can see that our model has achieved excellent performance in the COCO-Inpainted, KC-3D, and Synthtext-Change datasets compared with the CYWS-3D model, and its performance in other datasets is stable.

[0034] Example 3 In this embodiment, Figure 5As shown, initially image L and image R are taken from different perspectives. Due to the different perspectives, there will be occlusions between objects in the image. The purpose of this model is to capture significant changes in camera position and time, and we hope to locate the changes between them. In particular, we hope to capture everything that is physically different in the area visible in the two images, while ignoring areas that appear or disappear from view due to changes in camera pose or occlusion. This includes objects that may have been added or removed from the scene, as well as text or decorations that may have been added to the objects, while ignoring photometric differences, such as lighting changes. In addition, in order to ignore additional information such as camera parameters and camera pose, only RGB images are operated, so that changes between image pairs can be detected regardless of the scene. Figure 5 As shown, the object that changes between the two images is the shoe, which exists in image L but not in image R. Through the model, the different change areas of the image pair are finally detected and annotated with the detection box to obtain the segmented area.

[0035] Example 4 An image change segmentation device based on perspective alignment, comprising: Data acquisition unit: used to acquire the original image pair and preprocess the image pair to obtain two preprocessed images; 2D scene alignment unit: used for passing the two pre-processed images through the 2D scene alignment module to obtain an aligned image corresponding to the two images; Feature extraction unit: used to input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction to obtain feature information corresponding to the two images; A first construction unit is used to construct a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and feature information corresponding to two aligned images is input into the preliminary change detection network to obtain difference information corresponding to the two images; The second construction unit is used to construct a feature fusion module, and input the difference information corresponding to the two images into the feature fusion module for feature fusion to obtain the change information corresponding to the two images; The third construction unit is used to construct a positioning network for border detection, and input the change information corresponding to the two images into the positioning network for border detection to obtain the boundary boxes of the change areas of the two images respectively.

[0036] Example 5 An electronic device comprises: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the image change segmentation method based on perspective alignment as described above are performed.

[0037] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] Example 6 A computer-readable storage medium is characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the image change segmentation method based on perspective alignment as described above are executed.

[0039] Storage media include: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media that can store programs.

[0040] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for image change segmentation based on perspective alignment, characterized in that: The following steps are involved: S1. Obtain an original image pair, and preprocess the image pair to obtain two preprocessed images; S2. The two preprocessed images are passed through a 2D scene alignment module to obtain an aligned image corresponding to the two images; S3. Construct a feature extraction module, input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction, and obtain feature information corresponding to the two images; S4. constructing a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and inputting feature information corresponding to the two aligned images into the preliminary change detection network to obtain difference information corresponding to the two images; S5. Construct a feature fusion module, input the difference information corresponding to the two images into the feature fusion module for feature fusion, and obtain the change information corresponding to the two images; S6. Construct a positioning network for border detection, input the change information corresponding to the two images into the positioning network for border detection, and obtain the bounding boxes of the changed areas of the two images respectively.

2. The image change segmentation method based on view alignment according to claim 1, characterized in that: Step S1 is specifically as follows: Perform geometric transformation on the two original images and obtain the preprocessed size as Left image and right image , , Indicates that the data type of the matrix elements is real number, is the height of the image, is the width of the image, 3 is the number of channels of the image, Representing images It is a shape with a size of The real matrix of Same reason.

3. The image change segmentation method based on view alignment according to claim 2, characterized in that: The 2D scene alignment module in step S2 is composed of an image feature point matching unit and a homography registration unit, specifically: S21. Preprocessed left image and right image The image feature point matching unit uses the feature matching extractor Extract feature points and get the left image Matching feature point set and right image Matching feature point set , while the preprocessed left image and right image Using a monocular depth estimator Get the corresponding depth map respectively ; S22. The feature point set and After the homography registration unit calculates the homography transformation matrix, we get The transformation matrix ,and The transformation matrix , the transformation matrix and Apply to the corresponding image to achieve image alignment and get the left image Aligned images in 2D scenes based on the right image R and right image Aligned images in 2D scenes based on the left image L , .

4. The image change segmentation method based on view alignment according to claim 3, characterized in that: The feature extraction module in step S3 includes a baseline feature extraction model and a U-Net encoder, specifically: S31. The preprocessed left image , right image after preprocessing , left image Aligned images of and right image Aligned images of Input into the benchmark feature extraction model for feature extraction, and obtain the corresponding preliminary feature information , the preprocessed left image Corresponding preliminary feature information With the right image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the left image L , the preprocessed right image Corresponding preliminary feature information With the left image Aligned images of Corresponding preliminary feature information Merge by merging the number of channels to obtain the merged feature information of the right image R ; S32. and Input them into their respective U-Net compilers and output them respectively and Corresponding five intermediate feature maps of different scales and , When s=1, the scale of the feature map is 64×64, when s=2, the scale of the feature map is 32×32, when s=3, the scale of the feature map is 16×16, when s=4, the scale of the feature map is 8×8, and when s=5, the scale of the feature map is 4×4.

5. The image change segmentation method based on view alignment according to claim 4, characterized in that: The preliminary change detection network in step S4 includes a 3D scene image registration module and a difference module. The difference module is composed of a subtraction operation, specifically: S41. The corresponding intermediate feature map and The corresponding intermediate feature map Perform the channel half-division operation respectively to obtain the left image The corresponding feature map ,image The corresponding feature map , right image The corresponding feature map and images The corresponding feature map ,in, ; S42. In the 3D scene image registration module, the preprocessed left image is obtained according to step S21. The corresponding feature point set , right image The corresponding feature point set And the preprocessed left image The corresponding depth map And the right image The corresponding depth map , first for each point in the feature point set Perform back projection to get the projection vector , and then get the feature point set in homogeneous coordinates The corresponding sparse 3D point clouds , according to the sparse 3D point cloud through the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , and according to the sparse three-dimensional point cloud, the calculation formula , it is estimated Towards Aligned 3D linear transformation matrix , the formula is as follows: ， ， ， in, express The depth value of express The matrix generalized inverse of express The matrix generalized inverse of ; S43. The left image The corresponding feature map The image coordinates are used to obtain the 3D linear transformation matrix in step S42 Transform it into Image coordinates in three-dimensional coordinate system , use the point cloud tool to convert the converted image coordinates and feature map Encapsulate into a point cloud object, and finally render the point cloud object into a feature map through a differentiable renderer Aligning image features based on 3D scene correspondence , ; Similarly, we get the feature map Aligning image features based on 3D scene correspondence , ; S44. The left image The corresponding feature map ,image The corresponding feature map And feature map Aligning image features based on 3D scene correspondence Input into the difference module, and through the subtraction operation, the left image is obtained Based on the difference information in 2D scene and left image Based on the difference information in 3D scene , the formula is as follows: ， Similarly, the right image R is obtained based on the difference information in the 2D scene And the right image R based on the difference information in the 3D scene .

6. The image change segmentation method based on view alignment according to claim 5, characterized in that: The feature fusion module in step S5 is specifically: The difference information and Perform splicing to obtain splicing features, and input the splicing features into the linear layer to obtain intermediate features , and then perform convolution operation to get the weight , the weight With the left image The corresponding feature map Multiply element by element and finally get the left image Change information , the formula is as follows: ， ， ， in, Indicates that two features are merged into one channel. represents a nonlinear activation function, represents the first convolutional layer, represents the second convolutional layer, represents the Sigmoid activation function, Represents an element-by-element multiplication operation; similarly, we get the right image Change information .

7. The image change segmentation method based on view alignment according to claim 6, characterized in that: Step S6 is specifically as follows: The change information and After upsampling and decoding by U-Net decoder, the left image is obtained The corresponding feature map and right image The corresponding feature map , and then the feature map and The data is input into the CenterNet network head for border detection and positioning to obtain the bounding boxes of the changed areas of the two images.

8. An image change segmentation device based on perspective alignment, characterized in that: include: Data acquisition unit: used to acquire the original image pair and preprocess the image pair to obtain two preprocessed images; 2D scene alignment unit: used for passing the two pre-processed images through the 2D scene alignment module to obtain an aligned image corresponding to the two images; Feature extraction unit: used to input the two preprocessed images and the aligned images corresponding to the two images into the feature extraction module for feature extraction to obtain feature information corresponding to the two images; A first construction unit is used to construct a preliminary change detection network, wherein the preliminary change detection network includes a 3D scene image registration module and a difference module, and feature information corresponding to two aligned images is input into the preliminary change detection network to obtain difference information corresponding to the two images; The second construction unit is used to construct a feature fusion module, and input the difference information corresponding to the two images into the feature fusion module for feature fusion to obtain the change information corresponding to the two images; The third construction unit is used to construct a positioning network for border detection, and input the change information corresponding to the two images into the positioning network for border detection to obtain the boundary boxes of the change areas of the two images respectively.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the image change segmentation method based on perspective alignment as described in any one of claims 1 to 7 is performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image change segmentation method based on viewpoint alignment as claimed in any one of claims 1 to 7 is executed.

Images