Scene change detection method and device, electronic equipment and storage medium

Abstract

The invention provides a scene change detection method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring image pairs obtained by shooting a scene at different moments and at different visual angles, wherein the image pairs comprise a first image and a second image; performing image analysis on the first image and the second image by using a geometric basic model to obtain geometric priori knowledge; establishing a relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image by using the geometric priori knowledge to obtain a geometric relationship; and performing change prediction on the first image and the second image by using the geometrical relationship to obtain a final change mask. According to the invention, the robustness of scene change detection on visual angle difference and shielding can be improved under the condition that strict alignment in advance is not needed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for scene change detection. Background Technology

[0002] Scene change detection (SCD) has significant applications in tasks such as 3D scene updating, robot object-level navigation, and autonomous driving environmental perception. However, existing methods are mostly designed and evaluated under the assumption of roughly aligned viewpoints, which differs significantly from real-world complex scenes. Images acquired across timeframes often exhibit substantial viewpoint differences due to sensor installation positions, motion trajectories, and environmental disturbances, especially prevalent in autonomous driving, drone, and mobile robot scenarios. In summary, existing technologies suffer from poor robustness in scene change detection. Summary of the Invention

[0003] The main objective of this application is to provide a scene change detection method, apparatus, electronic device, and storage medium that can improve the robustness of scene change detection to viewpoint differences and occlusion without the need for strict pre-alignment.

[0004] To achieve the above objectives, a first aspect of this application proposes a scene change detection method, the method comprising: Acquire image pairs of a scene captured at different times and from different perspectives, wherein the image pairs include a first image and a second image; Image analysis is performed on the first and second images using a geometric fundamental model to obtain geometric prior knowledge; Using the aforementioned geometric prior knowledge, a relationship is established between the first pixel coordinates of the first image and the second pixel coordinates of the second image to obtain a geometric relationship; The geometric relationship is used to predict changes in the first image and the second image to obtain the final change mask.

[0005] Optionally, the step of performing scene analysis on the first image and the second image using a geometric foundation model to obtain geometric prior knowledge includes: Geometric information is obtained by using a geometric fundamental model to perform information reasoning on the first image and the second image; wherein, the geometric information includes matching pixel coordinate pairs; wherein, the matching pixel coordinate pairs include a first matching pixel coordinate in the first image and a second matching pixel coordinate in the second image; The geometric information is repaired using the matched pixel coordinates to obtain the geometric prior knowledge.

[0006] Optionally, the geometric information may also include intrinsic parameters of the first camera, intrinsic parameters of the second camera, extrinsic parameters of the first camera, extrinsic parameters of the second camera, and the relative scale transformation relationship between the first camera and the second camera; The step of using a geometric fundamental model to perform information reasoning on the first image and the second image to obtain geometric information includes: Using the aforementioned geometric foundation model, point cloud inference is performed on the image pairs to obtain the first self-point cloud and first self-confidence of the first image in the first camera coordinate system, the first interactive point cloud and first interactive confidence of the first image in the second camera coordinate system, the second self-point cloud and second self-confidence of the second image in the second camera coordinate system, the second interactive point cloud and second interactive confidence of the second image in the first camera coordinate system, and the matching pixel coordinate pairs. Based on the unknown first camera intrinsic parameters, the first self-point cloud, and the pixel space of the first image, a first linear equation is established, and the first camera intrinsic parameters are solved based on the first self-confidence and the first linear equation. Based on the unknown second camera intrinsics, the second self-point cloud, and the pixel space of the second image, a second linear equation is established, and the second camera intrinsics are solved based on the second self-confidence and the second linear equation. Based on the unknown first camera extrinsic parameters, the unknown second camera extrinsic parameters, the unknown scale transformation relationship, the first self-point cloud, the first interactive point cloud, the second self-point cloud, and the second interactive point cloud, a relative transformation equation is established. Based on the first self-confidence, the first interactive confidence, the second self-confidence, the second interactive confidence, and the relative transformation equation, the first camera extrinsic parameters, the second camera extrinsic parameters, and the relative scale transformation relationship are solved.

[0007] Optionally, the geometric prior knowledge further includes a first denoised depth map and a second denoised depth map; the step of repairing the geometric information using the matching pixel coordinate pairs to obtain the geometric prior knowledge includes: The coordinate scales of the first self-point cloud and the second self-point cloud are aligned using the relative scale transformation relationship. The last dimension data is extracted from the first point cloud to obtain the first depth map, and the last dimension data is extracted from the second point cloud to obtain the second depth map; The first depth map is converted into a first expression containing unknown first affine distortion parameters and an unknown first denoised depth map, and the second depth map is converted into a second expression containing unknown second affine distortion parameters and an unknown second denoised depth map. The first affine distortion parameter and the second affine distortion parameter are solved using the matched pixel coordinate pairs, the first self-confidence, the second self-confidence, the first expression, and the second expression to obtain the first denoised depth map and the second denoised depth map.

[0008] Optionally, the geometric prior knowledge includes a first depth map and a second depth map; the geometric prior knowledge is used to establish a relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image to obtain a geometric relationship; The first pixel coordinates in the first depth map are back-projected onto the first camera coordinate system to obtain the first back-projected point cloud. The first back-projected point cloud is transformed into the second camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the first transformed pixel coordinates. The first transformed pixel coordinates are projected onto the second depth map to obtain the first pixel projected coordinates. A first geometric relationship is generated based on the first pixel coordinates and the first pixel projected coordinates. The second pixel coordinates in the second depth map are back-projected onto the second camera coordinate system to obtain the second back-projected point cloud. The second back-projected point cloud is transformed into the first camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the second transformed pixel coordinates. The second transformed pixel coordinates are projected onto the first depth map to obtain the second pixel projection coordinates. A second geometric relationship is generated based on the second pixel coordinates and the second pixel projection coordinates.

[0009] Optionally, the geometric relationship includes a first geometric relationship and a second geometric relationship; the step of using the geometric relationship to perform change detection on the first image and the second image to obtain the final change mask includes: Based on the first geometric relationship and the second geometric relationship, the overlapping area between the first image and the second image is determined to obtain the first overlapping area and the first occlusion mask on the first image, and the second overlapping area and the second occlusion mask on the second image. The first image is encoded by multiple cascaded Transformer layers in the encoder to obtain the first key feature of the first Transformer layer and the first final embedding of the last Transformer layer. The second image is encoded by multiple cascaded Transformer layers in the encoder to obtain the second key feature of the first Transformer layer and the second final embedding of the last Transformer layer; For the first pixel in the overlapping region in the first image, a similarity calculation is performed based on the first key feature corresponding to the first pixel and the second key feature corresponding to the pixel that matches the first pixel in the second image to obtain a first feature similarity map; An adaptive threshold function is used to convert the first feature similarity map into a first initial change proposal, and a first occlusion mask is used to filter the first initial change proposal to obtain a first coarse change mask for the first image. For the second pixel in the overlapping region in the second image, a similarity calculation is performed based on the second key feature corresponding to the second pixel and the first key feature corresponding to the pixel that matches the second pixel in the first image to obtain a second feature similarity map; The second feature similarity map is converted into a second initial change proposal using an adaptive threshold function, and the second initial change proposal is filtered using a second occlusion mask to obtain a second coarse change mask for the second image. The first final embedding is decoded by the decoder to obtain the first category mask of the first image, and the second final embedding is decoded by the decoder to obtain the second category mask of the second image; The final change mask is obtained by performing mask fusion based on the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask.

[0010] Optionally, the step of mask fusion based on the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask to obtain the final change mask includes: The first fine change mask is obtained by performing mask matching based on the first coarse change mask and the first category mask; Then, based on the second coarse change mask and the second category mask, mask matching is performed to obtain the second fine change mask; The final change mask is obtained by geometrically fusing the first and second fine change masks.

[0011] To achieve the above objectives, a second aspect of this application provides a scene change detection device, the device comprising: An image acquisition module is used to acquire image pairs of a scene taken at different times and from different perspectives, the image pairs including a first image and a second image; The image analysis module is used to perform image analysis on the first image and the second image using the geometric foundation model to obtain geometric prior knowledge. The relationship establishment module is used to establish a relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image using the geometric prior knowledge, thereby obtaining a geometric relationship. The change prediction module is used to predict changes in the first image and the second image using the geometric relationship to obtain the final change mask.

[0012] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a scene change detection method as described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the scene change detection method described in the first aspect.

[0014] This application proposes a scene change detection method, apparatus, electronic device, and storage medium. It provides a technical concept that, during scene change detection, first generates geometric prior knowledge, then establishes geometric relationships, and finally generates the final change mask. This improves the robustness of scene change detection to viewpoint differences and occlusion without requiring strict pre-alignment. Furthermore, the entire process requires no additional training on any model, exhibiting training flexibility and good generalization characteristics.

[0015] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0016] Figure 1 This is a flowchart of a scene change detection method provided in an embodiment of this application; Figure 2 yes Figure 1 Flowchart for step 102; Figure 3 yes Figure 2 Flowchart of step 201; Figure 4 yes Figure 2 Flowchart for step 202; Figure 5 yes Figure 1 Flowchart for step 103; Figure 6 yes Figure 1 Flowchart for step 104; Figure 7 This is a schematic diagram of a scene change detection device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0021] Existing methods are mostly designed and evaluated under the assumption of basic viewpoint alignment, which differs significantly from real-world complex scenarios. In non-aligned viewpoint scenarios, mainstream SCD schemes have long relied on 2D appearance cues to establish cross-time correspondences and predict changes accordingly. Typical works include: DOF-CDNet, which uses optical flow for view alignment; CSCDNet, which introduces a correlation layer in the decoding stage to enhance cross-viewpoint robustness; and RSCD, which utilizes frozen DINO features and achieves full-image semantic alignment through cross-attention modules. These methods share the common characteristic of relying solely on 2D appearance features to estimate correspondences and then inputting the correlation into the change decoder. When the viewpoint difference increases, the appearance observation changes significantly, and optical flow or correlation matching is prone to drift or even failure. More importantly, these methods rely on limited 2D change mask supervision in small-scale SCD datasets, making it difficult to learn generalizable multi-view geometry knowledge. Consequently, they struggle to reliably determine visible overlap regions and explicitly model occlusion. These capabilities are difficult to acquire using only 2D information, resulting in significantly limited performance in complex scenes and under large viewpoint difference conditions.

[0022] In recent years, training-free (or zero-shot) approaches have alleviated the dependence on specialized data and training to some extent. For example, GeSCF systematically introduced the Segmentation Anything Model (SAM) into the zero-shot SCD framework for the first time, using a general segmentation prior and temporal consistency strategy to improve cross-domain generalization; ZSCD transfers the cross-frame consistency of tracking models to cross-time image pairs, generating variable segmentation under unlabeled conditions through style bridging and adaptive thresholding. These methods demonstrate that weakly supervised or even unsupervised SCD is feasible. However, their core remains focused on feature correlation and mask-level semantic consistency, lacking constraints on cross-view correspondence from a physical geometric perspective. Therefore, in scenarios with strong geometric changes and large occlusions, false positives of "invisibility equals change" are still prone to occur; the invisibility problem caused by large parallax or occlusion lacks explicit modeling. Experimental results show that the performance of zero-shot baseline methods degrades significantly when the viewpoint difference increases, and even almost fails in some difficult case groups.

[0023] Meanwhile, the development of Geometric Foundation Models (GFM) has provided a new technical path for unaligned viewpoint SCD. Represented by DUSt3R and MASt3R, GFM can directly feedforward regress pixel-level point clouds, depth, and camera parameters / 3D trajectories from a pair or more images without or with weak dependence on calibration and pose priors, achieving second-level 3D reconstruction. Such models significantly lower the threshold for geometric recovery, providing a solid foundation for cross-time pixel-level reprojection, overlap region determination, and occlusion inference. Combining geometric priors with visual foundation models such as SAM and DINO can elevate change detection from purely visual differences to a joint determination process of geometric consistency and semantic inconsistency: first, bidirectional pixel correspondence and a visible overlap region are obtained through geometric reprojection; then, appearance and semantic masks are compared within this region, and invisible differences caused by occlusion are explicitly removed, fundamentally improving robustness to large viewpoint differences and complex occlusions. Practice shows that the availability of explicit geometric priors and occlusion handling often determines the performance stability across datasets and scenes.

[0024] This application provides a scene change detection method, apparatus, electronic device, and storage medium, which are specifically described through the following embodiments. First, a scene change detection method in this application is described.

[0025] It should be noted that in each specific embodiment of this application, when it is necessary to process data related to user identity or characteristics, the user's permission or consent will be obtained first, and the collection, use and processing of such data will comply with relevant laws, regulations and standards.

[0026] Figure 1 This is an optional flowchart of a scene change detection method provided in the embodiments of this application, which may include, but is not limited to, steps 101 to 104.

[0027] Step 101: Obtain image pairs of the scene taken at different times and from different perspectives. The image pairs include a first image and a second image. Step 102: Perform image analysis on the first and second images using the geometric foundation model to obtain geometric prior knowledge; Step 103: Using prior geometric knowledge, establish the relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image to obtain the geometric relationship; Step 104: Perform change detection on the first and second images using geometric relationships to obtain the final change mask.

[0028] Steps 101 to 104 as illustrated in the embodiments of this application provide a technical concept for generating geometric prior knowledge, establishing geometric relationships, and finally generating the final change mask during scene change detection. This can improve the robustness of scene change detection to viewpoint differences and occlusion without the need for strict pre-alignment. Moreover, the entire process does not require additional training for any model, possessing training flexibility and good generalization characteristics.

[0029] In step 101 of some embodiments, "scene" refers to the object or environment being photographed. For example, the scene could be the ground or a kitchen. The first image is an image of the scene taken at a first moment and from a first perspective. The second image is an image of the scene taken at a second moment and from a second perspective. The first moment is earlier than the second moment. The first and second perspectives are different.

[0030] In step 102 of some embodiments, geometric prior knowledge is analyzed from the first image and the second image using a geometric foundation model.

[0031] In one embodiment, reference is made to Figure 2 Step 102 may include: Step 201: Use the geometric foundation model to perform information reasoning on the first image and the second image to obtain geometric information; wherein, the geometric information includes matching pixel coordinate pairs; wherein, the matching pixel coordinate pairs include the first matching pixel coordinate in the first image and the second matching pixel coordinate in the second image; Step 202: Use the matching pixel coordinate pairs to repair the geometric information and obtain geometric prior knowledge.

[0032] The advantage of this embodiment is that it can improve the accuracy of geometric prior knowledge, which is beneficial to improving the accuracy of scene change detection.

[0033] In step 201, the geometric information also includes the intrinsic parameters of the first camera (K1), the intrinsic parameters of the second camera (K2), the extrinsic parameters of the first camera (T1), the extrinsic parameters of the second camera (T2), and the relative scale transformation relationship between the first camera and the second camera. i ∈SE(3).

[0034] In one embodiment, reference is made to Figure 3 Step 201 may include: Step 301: Using the geometric basic model, point cloud inference is performed on the image pairs to obtain the first self point cloud and first self confidence of the first image in the first camera coordinate system, the first interactive point cloud and first interactive confidence of the second image in the second camera coordinate system, the second self point cloud and second self confidence of the second image in the second camera coordinate system, the second interactive point cloud and second interactive confidence of the second image in the first camera coordinate system, and the matching pixel coordinate pairs. Step 302: Based on the unknown first camera intrinsic parameters, the first self-point cloud, and the pixel space of the first image, establish the first linear equation, and solve the first camera intrinsic parameters based on the first self-confidence and the first linear equation. Step 303: Based on the unknown second camera intrinsic parameters, the second self-point cloud, and the pixel space of the second image, establish the second linear equation, and solve the second camera intrinsic parameters based on the second self-confidence and the second linear equation. Step 304: Based on the unknown first camera extrinsic parameters, the unknown second camera extrinsic parameters, the unknown scale transformation relationship, the first self point cloud, the first interactive point cloud, the second self point cloud, and the second interactive point cloud, establish a relative transformation equation. Based on the first self confidence level, the first interactive confidence level, the second self confidence level, the second interactive confidence level, and the relative transformation equation, solve for the first camera extrinsic parameters, the second camera extrinsic parameters, and the relative scale transformation relationship.

[0035] Specifically, record For size Image The pixel space. Representing an image width, Representing an image The height of the image. i represents the image index. Pair the given image with... Permutations and combinations are input into the GFM model (which has the ability to output pixel-wise 3D points and confidence scores) and inference is performed to obtain pixel-wise reconstructions of the first image. Second image at the first camera coordinates First self-point cloud in the second camera coordinate system First self-confidence First interactive point cloud First interaction confidence Second self-point cloud Second self-confidence Second interactive point cloud Second interactive confidence ,as well as For matching pixel coordinate pairs .in, Indicates the image Pixel-by-pixel correspondence Point cloud in camera coordinate system Indicates the image Pixel-by-pixel correspondence in the image Point cloud in camera coordinate system, For the first Matching pixel coordinate pairs in the first image pixel coordinates in For the first Matching pixel coordinate pairs in the second image The pixel coordinates in the perspective projection. Camera internal parameters Camera external parameters .for and The pixel-by-pixel correspondence, based on the projection relationship, has A question about camera internal parameters The linear equations (including the first and second linear equations): The number of equations is greater than the camera intrinsic parameters. The system of equations can be solved using least squares to determine the degrees of freedom of the camera, or other algorithms can be used to optimize the distance between the confidence-weighted projected pixel coordinates and the original pixel coordinates to recover the camera intrinsics. : Additionally, regarding and The pixel-by-pixel correspondence has the following based on spatial coordinate transformation: A relationship regarding relative scale transformation And the relationship of relative camera coordinate system transformation (hereinafter referred to as relative pose relationship). The equation (i.e., the equation of relative transformation): , .

[0036] in, Representing an image Transform to image The relative pose relationship. Representing an image external parameters of the camera Representing an image external parameters of the camera Representing an image Transform to image The rotation matrix, Representing an image Transform to image The translation vector. Note the symmetry of the transformation, we have A relationship regarding relative scale transformation Relative pose relationship The equation is given. This problem can be solved by optimizing the sum of squared distances between the transformed 3D coordinates using a confidence-weighted algorithm: .

[0037] Finally, the initialized camera extrinsic parameters can be obtained. Camera external parameters Relative scale transformation relationship .

[0038] The advantage of the embodiments of steps 301 to 304 described above is that they can obtain rich geometric information, which is beneficial to improving the accuracy of scene change detection.

[0039] In one embodiment, the geometric prior knowledge further includes a first denoised depth map and a second denoised depth map; refer to Figure 4 Step 202 may include: Step 401: Align the coordinate scales of the first self-point cloud and the second self-point cloud using the relative scale transformation relationship; Step 402: Extract the last dimension data from the first point cloud to obtain the first depth map, and extract the last dimension data from the second point cloud to obtain the second depth map; Step 403: Convert the first depth map into a first expression containing unknown first affine distortion parameters and unknown first denoised depth map, and convert the second depth map into a second expression containing unknown second affine distortion parameters and unknown second denoised depth map. Step 404: Solve for the first affine distortion parameter and the second affine distortion parameter using the matched pixel coordinate pairs, the first self-confidence, the second self-confidence, the first expression, and the second expression to obtain the first denoised depth map and the second denoised depth map.

[0040] Specifically, since the point cloud in the geometric information output by the geometric foundation model is often noisy, the previously obtained geometric information is not entirely accurate, especially regarding camera extrinsic parameters. Here, the matching pixel coordinate pairs obtained from the previous GFM model are used... Refine the geometric information. First, define the 3D coordinates and... Coordinate scale alignment, such as . This represents the relative scale transformation relationship between the coordinates in the second image and the first camera coordinate system of the first image. For point clouds in a camera coordinate system with consistent coordinate scale... The last dimension of data is directly taken as the depth map. Here we assume a depth map. Affine distortion exists, that is... ,in The first affine distortion parameters include The first affine distortion parameters include Here, reprojection errors in 3D can be used to optimize affine distortion parameters and fine-tune the initialized camera information:

[0041] .

[0042] in, This is to normalize the relative pose relationship during the optimization process. To prevent it from deviating excessively from its initial relative pose. ,in Regular rotation matrix Relative to the initial rotation matrix The square of the included angle, and Regular translation vector Relative to the initial translation vector The square of the distance is used as a tradeoff factor, which is generally selected. . This is to ensure that the depth after the affine transformation cannot be negative; therefore, the tradeoff coefficient here... We used larger values. After optimization, we obtained more accurate camera parameters (including relative pose relationships) and depth maps.

[0043] The advantage of the embodiments of steps 401 to 404 described above is that they optimize geometric information, which helps to improve the accuracy of scene change detection.

[0044] In step 103 of some embodiments, a geometric relationship is established by using geometric prior knowledge to establish the relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image. The geometric prior knowledge includes a first depth map and a second depth map.

[0045] In one embodiment, reference is made to Figure 5 Step 103 may include: Step 501: Back-project the first pixel coordinates in the first depth map to the first camera coordinate system to obtain the first back-projected point cloud. Transform the first back-projected point cloud to the second camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the first transformed pixel coordinates. Project the first transformed pixel coordinates to the second depth map to obtain the first pixel projected coordinates. Generate the first geometric relationship based on the first pixel coordinates and the first pixel projected coordinates. Step 502: Back-project the second pixel coordinates in the second depth map to the second camera coordinate system to obtain the second back-projected point cloud. Transform the second back-projected point cloud to the first camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the second transformed pixel coordinates. Project the second transformed pixel coordinates onto the first depth map to obtain the second pixel projection coordinates. Generate the second geometric relationship based on the second pixel coordinates and the second pixel projection coordinates.

[0046] Specifically, with pixel coordinates For example, first use Backproject it to The camera's coordinate system, and then the relative pose relationship. Transform to The camera coordinate system to which it belongs and projected onto The first geometric relation is obtained. : Second geometric relation It can be obtained using a similar method.

[0047] The advantage of the embodiments of steps 501 to 502 described above is that they enable the construction of the geometric relationship between the first image and the second image, which helps to improve the accuracy of scene change detection.

[0048] In step 104 of some embodiments, geometric relationships are used to perform change detection on the first image and the second image to obtain the final change mask.

[0049] In one embodiment, reference is made to Figure 6 Step 104 may include: Step 601: Based on the first geometric relationship and the second geometric relationship, determine the overlapping area between the first image and the second image to obtain the first overlapping area and the first occlusion mask on the first image, and the second overlapping area and the second occlusion mask on the second image. Step 602: Encode the first image by cascading multiple Transformer layers in the encoder to obtain the first key feature of the nth Transformer layer and the first final embedding of the last Transformer layer. Step 603: Encode the second image using multiple cascaded Transformer layers in the encoder to obtain the... The second key features of each Transformer layer and the second final embedding of the last Transformer layer; Step 604: For the first pixel in the overlapping area of ​​the first image, perform similarity calculation based on the first key feature corresponding to the first pixel and the second key feature corresponding to the pixel that matches the first pixel in the second image to obtain the first feature similarity map; Step 605: The first feature similarity map is converted into a first initial change proposal using an adaptive threshold function, and the first initial change proposal is filtered using a first occlusion mask to obtain a first coarse change mask for the first image. Step 606: For the second pixel in the overlapping area of ​​the second image, calculate the similarity based on the second key feature corresponding to the second pixel and the first key feature corresponding to the pixel that matches the second pixel in the first image to obtain the second feature similarity map; Step 607: The second feature similarity map is converted into a second initial change proposal using an adaptive threshold function, and the second initial change proposal is filtered using a second occlusion mask to obtain a second coarse change mask for the second image. Step 608: Decode the first final embedding according to the decoder to obtain the first category mask of the first image, and decode the second final embedding according to the decoder to obtain the second category mask of the second image; Step 609: Perform mask fusion based on the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask to obtain the final change mask.

[0050] In step 601, for example, the set of pixels that still fall within the effective field of the image after reprojection is defined as the first overlapping region. Second overlapping region It can also be obtained symmetrically. and This allows subsequent correlation and segmentation to be performed only within overlapping regions, reducing false alarms.

[0051] Regarding the first and second occlusion masks: To avoid misjudging areas visible only in a single frame as scene changes, an adaptive multi-criteria occlusion detection method combining depth residuals and confidence scores is adopted. For any... First calculate its position. 3D coordinates in the camera coordinate system And denote the corresponding reprojection depth as At the same time, take its place Geometric corresponding pixels Read the depth of the target view .definition Corresponding depth residual This method employs pixel-level adaptive thresholding. Screening of residuals, when At that time, it is considered that the pixel exist As can be seen, and in Areas that are occluded are denoted as occluded regions. To balance depth confidence, scene scale, and noise robustness, a threshold is set... The global baseline is determined by the following formula. Then it is used for pixel-level determination: .

[0052] for The two components, each playing a different role. Geometric scale term Use the second depth map correctly confidence level Using a weighted median statistic instead of a simple mean as the scaling benchmark ensures that the allowable error is linearly amplified in distant scenes, while avoiding excessively wide thresholds in close-up scenes. Furthermore, confidence-weighted scaling mitigates the impact of small amounts of noise, guaranteeing robustness across scene scales. (Statistical robustness terms) With confidence level Weighted residuals The median statistic is used as the center, and then a multiple of the weighted absolute median difference statistic is added to define abnormally large differences, thus automatically adapting to different noise levels and outlier ratios. Combined with robust statistics of confidence levels, it can automatically adapt to different noise levels and a small number of outliers. It is important to emphasize that... middle Scale factor Makes it possible for Medium geometric recovery is reliable but For unreliable pixels (e.g., occluded), the threshold will be tightened appropriately, making them more likely to be judged as occluded; while when the confidence levels of two views are similar, it degenerates into units. Threshold determination avoids over-reliance on the confidence level of a single view. In this embodiment, the parameter value is... Finally, the first occlusion mask of the first image is obtained. The second occlusion mask of the second image It can be obtained by swapping the view indexes and calculating in the same way.

[0053] In steps 602 to 603, for example, the image pairs are encoded using a SAM encoder. Encode them separately and extract the multi-head key (Key) features of the nth Transformer layer in the inference process. And the final layer embedding. . Indicates the first key feature, Indicates the second bond feature, Indicates the first final embedding, This indicates the second final embedding.

[0054] In step 604, the first key feature is upsampled to the image resolution using the first final embedding for pixel-level computation. Each pixel in the overlapping area With the first geometric relation The cosine similarity is calculated for the intermediate layer key features, i.e. The average first feature similarity map is obtained. .

[0055] In step 605, the first feature similarity map is then analyzed using an Adaptive Threshold Function (ATF). Conversion to the first initial change recommendation To filter out the second image The obscured area is then processed using the first obscuration mask. Screening is conducted as... First coarse change mask: .

[0056] The principle of step 606 is similar to that of step 604, and the principle of step 607 is similar to that of step 605, so they will not be described again here.

[0057] In step 608, the final embedded output class mask can be determined based on the SAM decoder. Specifically, the SAM decoder decodes the first final embedding to obtain the first category mask of the first image. Decoding the second final embedding yields the second category mask for the second image. ; Finally, in step 609, the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask are fused to obtain the final change mask. .

[0058] The advantage of the embodiments of steps 601 to 609 above is that by using the intermediate features and mask output of the encoder guided by geometric correspondence in the overlapping area to generate an initial change response, and by filtering and fusing the mask through geometric semantic consistency constraints and confidence weighting strategy to obtain the final change mask, the robustness of scene change detection is improved.

[0059] In one embodiment, step 609 may include: performing mask matching based on a first coarse change mask and a first category mask to obtain a first fine change mask; performing mask matching based on a second coarse change mask and a second category mask to obtain a second fine change mask; and performing geometric fusion based on the first fine change mask and the second fine change mask to obtain a final change mask.

[0060] Specifically, geometric-semantic mask matching (GSMM) is used in the first image. The first fine-tuning mask is obtained from the class-independent mask set output by SAM. Choose those that are with A first fine-grained transformation mask that has sufficient intersection and exhibits semantic inconsistency between the two graphs: The same process is applied to the second image. Symmetrical application can yield a second fine change mask. Then utilize the established geometric relationships. It can Mapped to query perspective and The final change mask is obtained by merging the two: .

[0061] The advantage of the above embodiments is that they improve the accuracy of generating the final change mask.

[0062] In summary, this application achieves the following beneficial effects: it proposes a three-stage, training-free method for detecting changes in unaligned image scenes. The three stages are: geometric prior generation, geometric relationship establishment, and change mask prediction. Specifically, firstly, a geometric foundation model with pixel-by-pixel 3D point and confidence output capabilities is used to infer camera intrinsic and extrinsic parameters, scale transformation relationships, and pixel-level point clouds and depth maps from two images from different perspectives and at different times. The geometric results are then refined using the model's built-in matching pairs and confidence information to obtain more stable geometric prior knowledge. Subsequently, pixel-level bidirectional geometric correspondences are established based on reprojection relationships to determine the visible overlapping region. Occlusion masks are then adaptively estimated using robust statistics based on depth residuals and confidence. Finally, within the overlapping region, the geometric correspondence guides the intermediate features and mask outputs of the general segmentation model SAM to generate initial change responses. Masks are then filtered and fused using geometric semantic consistency constraints and confidence weighting strategies to obtain the final change mask. The entire process requires no additional training for any model, exhibiting training freedom and good generalization characteristics.

[0063] Please see Figure 7 This application also provides a scene change detection device that can implement the above-described scene change detection method. Figure 7 This application provides a module structure block diagram of a scene change detection device, which includes: The image acquisition module 701 is used to acquire image pairs of a scene taken at different times and from different perspectives. The image pairs include a first image and a second image. Image analysis module 702 is used to perform image analysis on the first image and the second image using the geometric foundation model to obtain geometric prior knowledge; The relation establishment module 703 is used to establish a relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image using prior geometric knowledge, so as to obtain a geometric relationship. The change prediction module 704 is used to predict changes in the first image and the second image using geometric relationships to obtain the final change mask.

[0064] It should be noted that the specific implementation of the scene change detection device is basically the same as the specific implementation of the scene change detection method described above, and will not be repeated here.

[0065] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the scene change detection method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0066] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called and executed by the processor 801 to execute a scene change detection method according to an embodiment of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0067] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described scene change detection method.

[0068] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0069] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0070] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0071] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0073] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0074] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or the indirect coupling or communication connection between the apparatus or units may be electrical, mechanical, or other forms.

[0076] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for detecting scene changes, characterized in that, The method includes: Acquire image pairs of a scene captured at different times and from different perspectives, wherein the image pairs include a first image and a second image; Image analysis is performed on the first and second images using a geometric fundamental model to obtain geometric prior knowledge; Using the aforementioned geometric prior knowledge, a relationship is established between the first pixel coordinates of the first image and the second pixel coordinates of the second image to obtain a geometric relationship; The geometric relationship is used to predict changes in the first image and the second image to obtain the final change mask.

2. The method according to claim 1, characterized in that, The process of performing scene analysis on the first and second images using a geometric foundation model to obtain geometric prior knowledge includes: Geometric information is obtained by using a geometric fundamental model to perform information reasoning on the first image and the second image; wherein, the geometric information includes matching pixel coordinate pairs; wherein, the matching pixel coordinate pairs include a first matching pixel coordinate in the first image and a second matching pixel coordinate in the second image; The geometric information is repaired using the matched pixel coordinates to obtain the geometric prior knowledge.

3. The method according to claim 2, characterized in that, The geometric information also includes the intrinsic parameters of the first camera, the intrinsic parameters of the second camera, the extrinsic parameters of the first camera, the extrinsic parameters of the second camera, and the relative scale transformation relationship between the first camera and the second camera; The step of using a geometric fundamental model to perform information reasoning on the first image and the second image to obtain geometric information includes: Using the aforementioned geometric foundation model, point cloud inference is performed on the image pairs to obtain the first self-point cloud and first self-confidence of the first image in the first camera coordinate system, the first interactive point cloud and first interactive confidence of the first image in the second camera coordinate system, the second self-point cloud and second self-confidence of the second image in the second camera coordinate system, the second interactive point cloud and second interactive confidence of the second image in the first camera coordinate system, and the matching pixel coordinate pairs. Based on the unknown first camera intrinsic parameters, the first self-point cloud, and the pixel space of the first image, a first linear equation is established, and the first camera intrinsic parameters are solved based on the first self-confidence and the first linear equation. Based on the unknown second camera intrinsics, the second self-point cloud, and the pixel space of the second image, a second linear equation is established, and the second camera intrinsics are solved based on the second self-confidence and the second linear equation. Based on the unknown first camera extrinsic parameters, the unknown second camera extrinsic parameters, the unknown scale transformation relationship, the first self-point cloud, the first interactive point cloud, the second self-point cloud, and the second interactive point cloud, a relative transformation equation is established. Based on the first self-confidence, the first interactive confidence, the second self-confidence, the second interactive confidence, and the relative transformation equation, the first camera extrinsic parameters, the second camera extrinsic parameters, and the relative scale transformation relationship are solved.

4. The method according to claim 3, characterized in that, The geometric prior knowledge also includes a first denoised depth map and a second denoised depth map; the step of repairing the geometric information using the matched pixel coordinate pairs to obtain the geometric prior knowledge includes: The coordinate scales of the first self-point cloud and the second self-point cloud are aligned using the relative scale transformation relationship. The last dimension data is extracted from the first point cloud to obtain the first depth map, and the last dimension data is extracted from the second point cloud to obtain the second depth map; The first depth map is converted into a first expression containing unknown first affine distortion parameters and an unknown first denoised depth map, and the second depth map is converted into a second expression containing unknown second affine distortion parameters and an unknown second denoised depth map. The first affine distortion parameter and the second affine distortion parameter are solved using the matched pixel coordinate pairs, the first self-confidence, the second self-confidence, the first expression, and the second expression to obtain the first denoised depth map and the second denoised depth map.

5. The method according to any one of claims 1 to 4, characterized in that, The geometric prior knowledge includes a first depth map and a second depth map; the geometric prior knowledge is used to establish the relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image to obtain a geometric relationship; The first pixel coordinates in the first depth map are back-projected onto the first camera coordinate system to obtain the first back-projected point cloud. The first back-projected point cloud is transformed into the second camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the first transformed pixel coordinates. The first transformed pixel coordinates are projected onto the second depth map to obtain the first pixel projected coordinates. A first geometric relationship is generated based on the first pixel coordinates and the first pixel projected coordinates. The second pixel coordinates in the second depth map are back-projected onto the second camera coordinate system to obtain the second back-projected point cloud. The second back-projected point cloud is transformed into the first camera coordinate system using the relative pose relationship between the first camera and the second camera to obtain the second transformed pixel coordinates. The second transformed pixel coordinates are projected onto the first depth map to obtain the second pixel projection coordinates. A second geometric relationship is generated based on the second pixel coordinates and the second pixel projection coordinates.

6. The method according to any one of claims 1 to 4, characterized in that, The geometric relationship includes a first geometric relationship and a second geometric relationship; the step of using the geometric relationship to perform change detection on the first image and the second image to obtain the final change mask includes: Based on the first geometric relationship and the second geometric relationship, the overlapping area between the first image and the second image is determined to obtain the first overlapping area and the first occlusion mask on the first image, and the second overlapping area and the second occlusion mask on the second image. The first image is encoded by multiple cascaded Transformer layers in the encoder to obtain the first... The first key feature of each Transformer layer and the first final embedding of the last Transformer layer; The second image is encoded by multiple cascaded Transformer layers in the encoder to obtain the first... The second key features of each Transformer layer and the second final embedding of the last Transformer layer; For the first pixel in the overlapping region in the first image, a similarity calculation is performed based on the first key feature corresponding to the first pixel and the second key feature corresponding to the pixel that matches the first pixel in the second image to obtain a first feature similarity map; An adaptive threshold function is used to convert the first feature similarity map into a first initial change proposal, and a first occlusion mask is used to filter the first initial change proposal to obtain a first coarse change mask for the first image. For the second pixel in the overlapping region in the second image, a similarity calculation is performed based on the second key feature corresponding to the second pixel and the first key feature corresponding to the pixel that matches the second pixel in the first image to obtain a second feature similarity map; The second feature similarity map is converted into a second initial change proposal using an adaptive threshold function, and the second initial change proposal is filtered using a second occlusion mask to obtain a second coarse change mask for the second image. The first final embedding is decoded by the decoder to obtain the first category mask of the first image, and the second final embedding is decoded by the decoder to obtain the second category mask of the second image; The final change mask is obtained by performing mask fusion based on the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask.

7. The method according to claim 6, characterized in that, The step of fusing the first coarse change mask, the first category mask, the second coarse change mask, and the second category mask to obtain the final change mask includes: The first fine change mask is obtained by performing mask matching based on the first coarse change mask and the first category mask; Then, based on the second coarse change mask and the second category mask, mask matching is performed to obtain the second fine change mask; The final change mask is obtained by geometrically fusing the first and second fine change masks.

8. A scene change detection device, characterized in that, The device includes: An image acquisition module is used to acquire image pairs of a scene taken at different times and from different perspectives, the image pairs including a first image and a second image; The image analysis module is used to perform image analysis on the first image and the second image using the geometric foundation model to obtain geometric prior knowledge. The relation establishment module is used to establish a relationship between the first pixel coordinates of the first image and the second pixel coordinates of the second image using the geometric prior knowledge, thereby obtaining a geometric relationship. The change prediction module is used to predict changes in the first image and the second image using the geometric relationship to obtain the final change mask.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium storing at least one program, each program including instructions that, when executed by a terminal, cause the terminal to perform the method according to any one of claims 1 to 7.

Images