Image difference detection method, model training method, system, device and medium

Abstract

The application provides an image difference detection method, a model training method, a system, a device and a medium, and belongs to the technical field of anomaly detection. The image difference detection method comprises the following steps: obtaining a template image and a to-be-detected image; performing image fusion on the template image and the to-be-detected image to obtain a fused image; performing post-difference calculation on the template image and the to-be-detected image to obtain a difference image; inputting the template image, the to-be-detected image, the fused image and the difference image into a pre-trained multi-element end-to-end learning model respectively to obtain a difference region labeling binary image; determining a difference part showing an anomaly from the difference region labeling binary image to obtain an image difference detection result. The application can realize intelligent end-to-end detection of image differences, has strong generalization ability and noise resistance, can reduce labor costs, reduce missed detection, and improve the accuracy of image difference detection.

Description

Technical Field

[0001] This application relates to the field of anomaly detection technology, and in particular to an image difference detection method, model training method, system, device and medium. Background Technology

[0002] Image difference and change detection is a popular research topic in computer vision, with significant research and application value in industrial applications, medical image analysis, and hyperspectral image processing. Currently, image difference detection is mainly based on pixel-level or element / object-level differences, or performed using deep learning models. These methods either ignore spatial context information and generate significant salt-and-pepper noise, or focus only on the differences between two input images, lacking robustness to false differences. Therefore, they are prone to false positives and false negatives, resulting in low accuracy in image difference detection. Summary of the Invention

[0003] The main objective of this application is to propose an image difference detection method, model training method, system, device, and medium that can improve the accuracy of image difference detection.

[0004] To achieve the above objectives, a first aspect of this application proposes an image difference detection method, the method comprising: acquiring a template image and an image to be detected; fusing the template image and the image to be detected to obtain a fused image; performing post-difference calculation on the template image and the image to be detected to obtain a difference image; inputting the template image, the image to be detected, the fused image, and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary map of difference regions labeled with the difference regions labeled with the difference regions; and determining the difference regions that display abnormalities from the binary map of difference regions to obtain an image difference detection result.

[0005] In some embodiments, the multivariate end-to-end learning model is trained through the following steps: acquiring a template sample image and a sample image to be detected; fusing the template sample image and the sample image to be detected to obtain a fused sample image; performing post-difference calculation on the template sample image and the sample image to be detected to obtain a difference sample image; performing mask processing on the sample image to be detected to mark the different regions between the sample image to be detected and the template sample image, obtaining a binarized difference annotation map; inputting the template sample image, the sample image to be detected, the fused sample image, and the difference sample image into the multivariate end-to-end learning model respectively to obtain a binary sample map of the difference region annotation; calculating the total loss value of the model training based on the difference annotation map and the binary sample map of the difference region annotation, and adjusting the parameters of the multivariate end-to-end learning model based on the total loss value to obtain the trained multivariate end-to-end learning model.

[0006] In some embodiments, calculating the total loss value for model training based on the difference-annotated map and the difference region labeled binary sample map includes: establishing training sample pairs based on the differences between the template sample image and the sample image to be detected; performing a contrast loss calculation on the corresponding difference-annotated map and the difference region labeled binary sample map for each training sample to obtain a first loss value; calculating the distance between the difference-annotated map and the difference region labeled binary sample map to obtain a second loss value; and performing a weighted calculation based on the first loss value and the second loss value to obtain the total loss value for model training.

[0007] In some embodiments, the step of calculating a first loss value by performing a contrast loss calculation on the difference-annotated map and the difference region-annotated binary sample map corresponding to each training sample pair includes: obtaining the number of training sample pairs; obtaining the difference between the template sample image and the sample image to be detected in each training sample pair to obtain sample labels representing samples that are the same or matched; obtaining the Euclidean distance between the difference-annotated map and the difference region-annotated binary sample map corresponding to each training sample pair to obtain sample Euclidean distance; and performing a contrast loss calculation on the difference-annotated map and the difference region-annotated binary sample map corresponding to each training sample pair based on the number of sample pairs, the sample labels, and the sample Euclidean distance to obtain the first loss value.

[0008] In some embodiments, before performing image fusion between the template image and the image to be detected, the method further includes: performing feature detection on the template image and the image to be detected, and matching feature points between the template image and the image to be detected; calculating the feature point similarity between the template image and the image to be detected, and sorting them by similarity to obtain a sorting result, and determining multiple target feature points from the sorting result; performing an affine transformation on the image to be detected based on the target feature points to obtain an image-registered image to be detected, wherein the image-registered image to be detected is used for image fusion and post-difference calculation with the template image.

[0009] In some embodiments, the step of fusing the template image and the image to be detected to obtain a fused image includes: reducing the transparency of the template image to obtain a first image; reducing the transparency of the image to be detected to obtain a second image; and performing weighted processing based on the first image and the second image to obtain the fused image.

[0010] In some embodiments, the step of inputting the template image, the image to be detected, the fused image, and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary image with labeled difference regions includes: inputting any one of the template image, the image to be detected, the fused image, and the difference image into the pre-trained tuple end-to-end learning model, and first splitting the input image into an average to obtain a first image block; setting the vector dimension of the first image block to a preset vector dimension value, and then performing a straightening operation to process the two-dimensional image data into a one-dimensional sequence to obtain a second image block; inputting the second image block into a preset plurality of Transformer modules for multiple information extraction and class pooling processes to obtain a third image block; stacking the third image blocks corresponding to the template image, the image to be detected, the fused image, and the difference image respectively by sharing feature extraction parameters, and applying a linear layer to compress them to the original size in terms of dimension to obtain a fourth image block; and upsampling the fourth image block to obtain a binary image with labeled difference regions.

[0011] In some embodiments, upsampling the fourth image block to obtain a binary image with labeled difference regions includes: inputting the fourth image block into a first linear layer, increasing the feature dimension to twice the original dimension, then using a rearrangement operation to expand the resolution of the input features to twice the input resolution, and reducing the feature dimension to half the input dimension to obtain a fifth image block; inputting the fifth image block into a preset plurality of Transformer modules and a block expansion layer for recovery processing to obtain a sixth image block; and inputting the sixth image block into a second linear layer to obtain a binary image with labeled difference regions.

[0012] In some embodiments, determining the abnormally displaying difference portion from the binary image labeled with the difference region to obtain an image difference detection result includes: reducing the transparency of the fused image to obtain a third image; reducing the transparency of the binary image labeled with the difference region to obtain a fourth image; performing weighted processing on the third image and the fourth image to obtain a fused target difference image; detecting the highlighted portion in the target difference image and determining that the highlighted portion is the abnormally displaying difference portion to obtain an image difference detection result.

[0013] To achieve the above objectives, a second aspect of this application proposes a model training method, the method comprising: acquiring a template sample image and a sample image to be detected; performing image fusion on the template sample image and the sample image to be detected to obtain a fused sample image; performing post-difference calculation on the template sample image and the sample image to be detected to obtain a difference sample image; performing mask processing on the sample image to be detected to mark the different regions between the sample image to be detected and the template sample image to obtain a binarized difference annotation map; inputting the template sample image, the sample image to be detected, the fused sample image, and the difference sample image into a multivariate end-to-end learning model to obtain a difference region annotation binary sample map; calculating the total loss value of model training based on the difference annotation map and the difference region annotation binary sample map, and adjusting the parameters of the multivariate end-to-end learning model based on the total loss value to obtain the trained multivariate end-to-end learning model.

[0014] To achieve the above objectives, a third aspect of this application proposes an image difference detection system, the system comprising: an image sampling module for acquiring a template image and an image to be detected; a fusion module for fusing the template image and the image to be detected to obtain a fused image; a difference module for performing post-difference calculation on the template image and the image to be detected to obtain a difference image; a model processing module for inputting the template image, the image to be detected, the fused image, and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary map of difference regions labeled with differences; and a difference determination module for determining the difference portions that display abnormalities from the binary map of difference regions labeled with differences to obtain an image difference detection result.

[0015] To achieve the above objectives, a fourth aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the image difference detection method described in the first aspect embodiment and the model training method described in the second aspect embodiment.

[0016] To achieve the above objectives, a fifth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the image difference detection method described in the first aspect embodiment and the model training method described in the second aspect embodiment.

[0017] The image difference detection method, model training method, system, device, and medium proposed in this application can be applied to an image difference detection system. This application first acquires a template image and a target image. After image fusion and post-difference calculation, a fused image and a difference image are obtained respectively. During difference detection, the template image, the target image, the fused image, and the difference image are input into a pre-trained tuple end-to-end learning model to obtain a binary map of the difference region annotations output by the model. This allows the identification of abnormally displayed difference parts from the map, resulting in the image difference detection result. This application learns image pseudo-differences and difference features in image pairs through four parallel streams, exhibiting strong learning ability for varying regions at different scales. It enables intelligent end-to-end detection of image pair differences, demonstrating strong generalization and noise resistance, reducing labor costs, minimizing false positives and false negatives, and improving the accuracy of image difference detection. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of the image difference detection method provided in the embodiments of this application;

[0019] Figure 2 This is a flowchart illustrating the training process of the multi-group end-to-end learning model provided in the embodiments of this application;

[0020] Figure 3 yes Figure 2 A flowchart illustrating step S206 in the process;

[0021] Figure 4 yes Figure 3 A flowchart illustrating step S301 in the process;

[0022] Figure 5 yes Figure 1 A flowchart illustrating the process prior to step S102;

[0023] Figure 6 yes Figure 1 A flowchart illustrating step S102 in the process;

[0024] Figure 7 yes Figure 1 A flowchart illustrating step S104 in the process;

[0025] Figure 8 yes Figure 7 A flowchart illustrating step S705 in the process;

[0026] Figure 9 yes Figure 1 A flowchart illustrating step S105 in the process;

[0027] Figure 10 This is a flowchart illustrating the model training method provided in the embodiments of this application;

[0028] Figure 11 This is a schematic diagram of the functional modules of the image difference detection system provided in the embodiments of this application;

[0029] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0033] First, let's analyze some of the terms used in this application:

[0034] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0035] Image difference and change detection is a popular research topic in the field of computer vision. Its goal is to build models using existing samples to detect and label various possible abnormal images. It has high research significance and application value in fields such as industrial appearance defect detection, medical image analysis, and hyperspectral image processing, such as packaging material design drafts, film difference review, product appearance defect detection, and remote sensing image change detection.

[0036] Depending on the research objective, traditional image difference detection methods can be divided into two categories: pixel-based and element- or object-based difference detection. Pixel-based difference detection generates a difference map by directly comparing the spectral or texture information of pixels, and then produces the final result image through thresholding or clustering. While this method is easy to implement, it ignores spatial context information and generates a large amount of salt-and-pepper noise. Element-based difference detection, on the other hand, divides the image into unconnected elements and analyzes the differences between images using spectral, texture, structural, and geometric information. Traditional methods in this area include Covariance Vector Analysis (CVA), Principal Component Analysis (PCA), and Multivariate Alteration Detection (MAD).

[0037] Moreover, due to the powerful learning and expression capabilities of deep learning, deep learning models such as convolutional neural networks are widely used in general computer vision tasks and have shown significant potential in image difference detection. However, existing models only focus on the differences between two input images and lack robustness to pseudo-difference information.

[0038] Based on this, embodiments of this application provide an image difference detection method, model training method, system, device, and medium. It can learn image pseudo-differences and difference features in image pairs through four parallel streams, exhibiting strong learning ability for regions of change at different scales. It can achieve intelligent end-to-end detection of image pair differences, with strong generalization ability and noise resistance. It can reduce the labor cost of manual detection due to low accuracy of image anomaly detection, reduce false positives and false negatives, and improve the accuracy of image difference detection.

[0039] The image difference detection method, model training method, system, device and medium provided in the embodiments of this application are specifically described through the following embodiments. First, the image difference detection method in the embodiments of this application is described.

[0040] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0041] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0042] The image difference detection method provided in this application relates to the field of artificial intelligence technology. This image difference detection method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the image difference detection method, but is not limited to the above forms.

[0043] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0044] It should be noted that in various specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. For example, when obtaining user-stored data and user cached data access requests, user permission or consent is obtained first; when obtaining samples or various image data, this application embodiment obtains user permission or consent first. Moreover, the collection, use, and processing of this data comply with the relevant laws, regulations, and standards of relevant countries and regions. Furthermore, when this application embodiment needs to obtain sensitive personal information of users, it obtains separate permission or consent from users through pop-ups or redirection to confirmation pages. Only after obtaining separate permission or consent from users is the necessary user-related data for the normal operation of this application embodiment obtained.

[0045] Figure 1This is an optional flowchart of the image difference detection method provided in the embodiments of this application. Figure 3 The method may include, but is not limited to, steps S101 to S105.

[0046] Step S101: Obtain the template image and the image to be detected;

[0047] For example, the image difference detection method can be applied in an image difference detection system (hereinafter referred to as the system). In the process of image difference detection, the system needs to acquire a template image and an image to be detected.

[0048] The template image is an image used as a standard template, also called a template image. The image to be detected is a target image used for image difference detection. In this embodiment, by acquiring the template image and the image to be detected, these two images can be processed to detect whether there are differences between the image to be detected and the template image, thereby finally completing the image difference detection.

[0049] It is understood that in the embodiments of this application, multiple template images or multiple images to be detected can be obtained. There can be a one-to-one correspondence between the template images and the images to be detected, that is, one template image corresponds to one image to be detected. In addition, there can be a one-to-many correspondence between the template images and the images to be detected, that is, one template image corresponds to multiple images to be detected. This embodiment of the application does not impose specific limitations on this.

[0050] In this embodiment of the application, by acquiring a template image and an image to be detected, an image pair can be established, which can then be used to detect the differences between the image pairs.

[0051] Step S102: The template image and the image to be detected are fused to obtain the fused image;

[0052] For example, in this embodiment of the application, image fusion is required between the template image and the image to be detected to obtain a fused image, which can then be used for subsequent difference detection. It should be noted that the fused image can also be understood as a synthetic image. The fused image can highlight some differences while maintaining the color features after image processing, such as highlighting element displacement, missing elements, size consistency, etc. between the two images before fusion.

[0053] Step S103: Perform post-difference calculation on the template image and the image to be detected to obtain the difference image;

[0054] For example, in this embodiment, post-difference calculation is performed on the template image and the image to be detected to obtain a difference image, which can then be used for subsequent difference detection. It should be noted that the difference image is obtained after post-difference calculation between the two images and contains important key information for difference detection, especially elemental differences. Because direct subtraction causes the difference image to lose image details contained in the original image, this embodiment utilizes the difference image in subsequent models to retain sensitivity to false differences, solve noise problems, and reduce misjudgments of false difference information.

[0055] Step S104: Input the template image, the image to be detected, the fused image and the difference image into the pre-trained multi-group end-to-end learning model to obtain the binary map of the difference region annotation.

[0056] For example, in this embodiment of the application, a multi-tuple end-to-end learning model (hereinafter referred to as the model) can be pre-trained. This model is a neural network model that can be pre-set in the system and used for image difference detection. In this embodiment of the application, the data input to the multi-tuple end-to-end learning model includes four parts: a template image, the image to be detected, a fused image, and a difference image, which together form a multi-tuple image. Inputting four images simultaneously can effectively improve the performance of the model.

[0057] As is understandable, a tuple-based end-to-end learning model is an end-to-end model; it takes image data as input and outputs an image. A tuple-based end-to-end learning model can be trained on multiple sample datasets, thereby improving its ability to process input tuple data after training and outputting more accurate binary maps with labeled difference regions.

[0058] It should be noted that, as described above, the template image, the image to be detected, the fused image, and the difference image in this application embodiment reflect different image features, thus generating less redundant information during the calculation process. For example, the model in this application embodiment can use a self-attention mechanism layer based on convolution-like structures to construct hierarchical feature representations through small image patches and layer-by-layer neighborhood merging, enabling the extraction of visual features at different levels, thereby improving the accuracy of anomaly detection. No specific limitations are imposed here.

[0059] After processing the input data, the tuple-based end-to-end learning model can output a binary map with labeled difference regions. This binary map is used to determine which elements in the image are different.

[0060] Step S105: Determine the abnormal difference portion from the binary map labeled with the difference region to obtain the image difference detection result.

[0061] For example, in the embodiments of this application, after obtaining the binary map of the difference region annotation output by the model, since the binary map can determine which element in the image is different, the difference part that shows anomalies can be determined from the binary map of the difference region annotation. This difference part is the part that is different between the image to be detected and the template image, and thus the image difference detection result can be obtained based on the difference part.

[0062] It is understandable that image difference detection results can characterize and indicate the differences between the image to be detected and the template image. After obtaining the image difference detection results, the user can understand the differences between the images. Alternatively, the image difference detection results can be sent to other terminal devices for parsing. After receiving the image difference detection results, the terminal devices can parse the results to obtain the differences between the image to be detected and the template image.

[0063] This application embodiment learns image pseudo-differences and difference features in image pairs through four parallel streams, exhibiting strong learning ability for regions of change at different scales. It can realize intelligent end-to-end detection of image pair differences, with strong generalization ability and noise resistance, which can reduce labor costs, reduce false detections and false negatives, and improve the accuracy of image difference detection.

[0064] Please see Figure 2 In some embodiments, the tuple end-to-end learning model is trained through the following steps, which may include steps S201 to S206:

[0065] Step S201: Obtain the template sample image and the sample image to be detected;

[0066] Step S202: The template sample image and the sample image to be detected are fused to obtain the fused sample image;

[0067] Step S203: Perform post-difference calculation on the template sample image and the sample image to be detected to obtain the difference sample image;

[0068] Step S204: Perform mask processing on the image to be detected, and mark the different regions between the image to be detected and the template image to obtain a binarized difference annotation map;

[0069] Step S205: Input the template sample image, the sample image to be detected, the fused sample image and the difference sample image into the multivariate end-to-end learning model respectively to obtain the binary sample image of the difference region annotation;

[0070] Step S206: Calculate the total loss value of the model training based on the difference annotation map and the binary sample map of the difference region annotation, and adjust the parameters of the multivariate end-to-end learning model according to the total loss value to obtain the trained multivariate end-to-end learning model.

[0071] For example, in this embodiment of the application, a tuple-based end-to-end learning model can be pre-trained. During the training process, this embodiment of the application can acquire multiple sample data, including template sample images and sample images to be detected.

[0072] The template sample image is an image used as a standard template, similar to the template image in the above application process embodiment. The sample image to be detected is a target image used for image difference detection during training, similar to the template image in the above application process embodiment. In this application embodiment, by acquiring the template sample image and the sample image to be detected, these two images can be processed to detect whether there is a difference between the sample image to be detected and the template sample image, thereby finally completing the training of the model.

[0073] It is understood that in the embodiments of this application, multiple template sample images or multiple sample images to be detected can be obtained. The template sample images and sample images to be detected can be in a one-to-one correspondence or a one-to-many correspondence, which will not be elaborated here.

[0074] In this embodiment, by acquiring a template image and an image to be detected, image pairs can be established, which can then be used to detect the differences between the image pairs. As training samples, the sample pairs consisting of the template sample image and the image to be detected can contain multiple differing image pairs and indistinguishable image pairs, so that the model can be trained for different difference situations.

[0075] For example, in this embodiment of the application, image fusion is required between the template sample image and the sample image to be detected to obtain a fused sample image, which can then be used for training subsequent difference detection. It should be noted that the fused sample image can also be understood as a synthetic image. Similar to the fused image in the above application process, the fused sample image can highlight some differences while maintaining the color features after image processing, such as highlighting element displacement, missing elements, size consistency, etc. between the two images before fusion.

[0076] For example, in this embodiment, post-difference calculation is performed on the template sample image and the sample image to be detected to obtain a difference sample image, which can then be used for training subsequent difference detection. It should be noted that the difference sample image is obtained after post-difference calculation between the two images. Similar to the difference image in the above application process, it contains important key information for difference detection, especially elemental differences. Because the direct subtraction operation causes the difference image to lose the image details contained in the original image, this embodiment utilizes the difference image in subsequent models to retain sensitivity to false differences, solve noise problems, and reduce misjudgments of false difference information.

[0077] For example, in order to train the model, it is necessary to establish a comparison between the model's output results to verify whether the model's output results are accurate. Therefore, in this embodiment, the image to be detected is masked to mark the different regions between the image to be detected and the template image, thus obtaining a binarized difference annotation map.

[0078] Specifically, the difference annotation map established in this embodiment is a binary segmentation map. Specifically, in the masking process, this embodiment uses black pixels as the base, with the same size as the template image, and uses a white mask to mark the different regions between the sample image to be detected and the template sample image, that is, a 0 / 1 binary segmentation map, thereby obtaining the difference annotation map.

[0079] In this embodiment, the data input to the multivariate end-to-end learning model for training includes four elements: a template sample image, a sample image to be detected, a fused sample image, and a difference sample image, forming a multivariate image set. Inputting four images simultaneously effectively improves the model's performance. The multivariate end-to-end learning model can be trained using multiple sample data sets, thereby improving its ability to process input multivariate data after training and outputting more accurate binary maps with labeled difference regions.

[0080] It should be noted that, as described above, the template image, the image to be detected, the fused image, and the difference image in this application embodiment reflect different image features, thus generating less redundant information during the calculation process. For example, the model in this application embodiment can use a self-attention mechanism layer based on convolution-like structures to construct hierarchical feature representations through small image patches and layer-by-layer neighborhood merging, enabling the extraction of visual features at different levels, thereby improving the accuracy of anomaly detection. No specific limitations are imposed here.

[0081] During training, the multivariate end-to-end learning model processes the input sample data and outputs a binary sample map with labeled difference regions. This binary map determines which elements in the image are different. During training, the total training loss is calculated based on the labeled difference map and the binary sample map, and the parameters of the multivariate end-to-end learning model are adjusted accordingly to obtain the trained multivariate end-to-end learning model.

[0082] It is understood that during the training process, the template sample image, the sample image to be detected, the fused sample image and the difference sample image can be obtained directly. That is, the training process may not perform image fusion and post-difference calculation processing on the template sample image and the sample image to be detected in the system, but directly obtain four sample images and establish tuple data. No specific restrictions are made here.

[0083] Please see Figure 3 In some embodiments, step S206 may include steps S301 to S303:

[0084] Step S301: Based on the difference between the template sample image and the sample image to be detected, establish training sample pairs, and perform contrast loss calculation based on the corresponding difference annotation map and difference region annotation binary sample map of each training sample pair to obtain the first loss value.

[0085] Step S302: Calculate the distance between the difference-annotated map and the difference region-annotated binary sample map to obtain the second loss value;

[0086] Step S303: Calculate the total loss value for model training by weighting the first loss value and the second loss value.

[0087] For example, in the sample pairs formed by the sample image and the sample image to be detected in the embodiments of this application, there are samples that are different from each other and samples that are not different from each other. This forms multiple training sample pairs based on different differences.

[0088] In order for the model to effectively distinguish between differential sample pairs and indifferent sample pairs, as well as the binary map of the differential region, this embodiment of the application uses contrastive loss and supervised loss of binary map of labeled data to establish the total loss value for model training.

[0089] Deep metric learning involves training a network to learn a non-linear transformation from the input to the embedding space. Samples with high similarity have shorter embedding vectors, and vice versa. Therefore, a contrastive loss is used to minimize the distance between indifferent pixel pairs and maximize the distance between differing pixel pairs in the embedding space. In the encoder Siamese neural network of this application, the loss function used is contrastive loss. This loss function can effectively handle the relationships between data pairs in the Siamese neural network, and the resulting loss value is a contrastive loss value, which is used as the first loss value for subsequent calculations.

[0090] Meanwhile, in order to fuse and decode the feature vectors extracted from the tuple into a difference binary map, a supervised learning method is used in this embodiment to calculate the distance between the labeled data and the decoded data. The labeled data is the difference labeled map, and the decoded data is the difference region labeled binary sample map generated by the model. The above is to calculate the distance between the difference labeled map and the difference region labeled binary sample map through supervised learning to obtain the second loss value for subsequent calculation.

[0091] After obtaining the first loss value and the second loss value, this embodiment of the application performs a weighted calculation based on the first loss value and the second loss value to obtain the total loss value for model training. It can be understood that in the process of weighted calculation to obtain the total loss value, this embodiment of the application can assign different weights to the first loss value and the second loss value respectively, and during the training process, the weights can be appropriately adjusted according to the training result, that is, the magnitude of the total loss value, to improve the computational power of the model.

[0092] Please see Figure 4 In some embodiments, step S301 may include steps S401 to S404:

[0093] Step S401: Obtain the number of training sample pairs;

[0094] Step S402: Obtain the differences between the template sample image and the sample image to be detected in each training sample pair to obtain sample labels that represent samples that are the same or matched.

[0095] Step S403: Obtain the Euclidean distance between the corresponding difference annotation map and the difference region annotation binary sample map for each training sample pair, and obtain the sample Euclidean distance;

[0096] Step S404: Based on the number of sample pairs, sample labels, and sample Euclidean distance, perform contrast loss calculation on the difference annotation map and difference region annotation binary sample map corresponding to each training sample pair to obtain the first loss value.

[0097] For example, in this embodiment of the application, the first loss value is represented by L1. In this embodiment of the application, an expression for the contrastive loss can be established, such as:

[0098]

[0099] Where N is the number of training sample pairs, i.e., the number of sample pairs; y is the sample label indicating whether two images match, used to characterize the difference between the template sample image and the sample image to be detected in each training sample pair. For example, y = 1 represents that the two samples are the same or match, and y = 0 represents that the two samples are different. It can be understood that the sample labels can be labeled when creating the image pairs, and no specific restrictions are made here; d = ||a n -b n || 2 , representing the Euclidean distance between two sample features, is also known as the sample Euclidean distance, where a n b is the feature vector of the difference-annotated map. n The feature vectors of the binary sample map are labeled for the difference region; moreover, in the above formula (1), margin is the set threshold, which can be pre-configured. The pre-selection of the threshold can be the default, and the threshold can be adjusted according to the training situation of the model. No specific restrictions are made here.

[0100] Meanwhile, in this embodiment, in order to fuse and decode the feature vectors extracted from the multivariate array into a difference binary map, a supervised learning method is adopted to calculate the distance between the labeled data and the decoded data. The smaller the distance, the better. The obtained second loss value is represented by L2. In this embodiment, a supervised learning expression can be established, such as:

[0101] L2=‖ground truth-prediction‖ 2 (2)

[0102] Here, ground truth represents the true value of the difference-annotated map, and prediction represents the true value of the difference-annotated binary sample map obtained by the model decoding.

[0103] Combining the first and second loss values, we obtain the total loss function Loss as follows:

[0104] Loss=ω1L1+ω2L2 (3)

[0105] Here, ω1 and ω2 are custom loss weights. For example, the initial weights can be set to 0.5 by default, and can be adjusted later based on the training results. In this embodiment, the model parameters are iteratively optimized with the goal of minimizing the total loss function value. The model with ideal performance is obtained and saved, and the trained model can be used in image difference detection applications.

[0106] Please see Figure 5 In some embodiments, steps S501 to S503 may be included before step S102:

[0107] Step S501: Perform feature detection on the template image and the image to be detected, and match the feature points between the template image and the image to be detected;

[0108] Step S502: Calculate the feature point similarity between the template image and the image to be detected, sort the similarity by size to obtain the sorting result, and determine multiple target feature points from the sorting result;

[0109] Step S503: Perform an affine transformation on the image to be detected based on the target feature points to obtain the image to be detected after image registration. The image to be detected after image registration is used for image fusion and post-difference calculation with the template image.

[0110] For example, in the application stage of this application, the acquired template image and the image to be detected can be preprocessed. Specifically, in the application, an image preprocessing module can be established in the system, and after the template image and the image to be detected are acquired, they are input into the preprocessing module. The system performs feature extraction, feature matching, and conversion registration on the template image and the image to be detected through the preprocessing module.

[0111] In this embodiment, feature detection is achieved through feature extraction and feature pairing, and the Speeded Up Robust Features (SURF) method is employed. SURF is a robust image feature recognition and description algorithm. SURF uses the determinant value of the Hessian approximation matrix for feature point response detection and accelerates computation using integral images. SURF's feature descriptors are based on 2D discrete wavelet transform response Haar wavelets and effectively utilize integral images. The basic steps of feature extraction and feature pairing using the SURF algorithm can be divided into four main parts: scale space establishment, feature point localization, feature point orientation determination, and feature point description.

[0112] Subsequently, in this embodiment, feature point matching is used to calculate the feature point similarity between the template image and the image to be detected, and the similarity is sorted to obtain the sorting result. The most relevant feature points are found and used as target feature points to solve the affine matrix M. The original image is then subjected to affine transformation through the matrix M to obtain the registered image to be detected. The size of the registered image to be detected is the same as that of the template image, which facilitates subsequent fusion processing and post-difference calculation.

[0113] It should be noted that the above affine transformation is a linear transformation that preserves the "parallelism" and "flatness" of the image. That is, the original straight lines and parallel lines in the image will still be preserved after the transformation. The key to performing the affine transformation is to solve for the affine matrix M, and then use it to transform the original image to obtain the final image, thereby obtaining the registered image to be detected in the embodiment of this application.

[0114] It is understood that image registration is also required during the model training stage of image difference detection in this application embodiment. For example, after obtaining the sample image to be detected, image registration of the template sample image is required, which will not be elaborated here.

[0115] Please see Figure 6 In some embodiments, step S102 may include steps S601 to S603:

[0116] Step S601: Reduce the transparency of the template image to obtain the first image;

[0117] Step S602: Reduce the transparency of the image to be detected to obtain the second image;

[0118] Step S603: Perform weighted processing on the first image and the second image to obtain the fused image.

[0119] For example, image fusion in this embodiment is achieved by reducing the transparency of the two images to be fused. Specifically, in this embodiment, the transparency of the template image can be reduced to obtain a first image, and the transparency of the image to be detected can be reduced to obtain a second image. After fusion, the first and second images obtained by reducing transparency will not cause the other image to be unclear. It can highlight some differences, such as element displacement, missing elements, and size consistency, while maintaining the color features. Then, the first and second images are weighted to obtain the fused image.

[0120] It is understood that in this embodiment, the transparency ratios of the template image and the image to be detected can be set according to the training effect, so that the synthesized fused image can highlight the differences while maintaining the color features. By default, in this embodiment, the transparency of the template image and the registered image to be detected is reduced by 50% simultaneously, and then synthesized into a single image, i.e., a weighted summation with each image having a weight of 0.5, to obtain the desired fused image. The transparency can be adjusted subsequently according to the training effect, such as reducing the transparency of one image to above 50% and the other to below 50%.

[0121] It should be noted that the image fusion process in the model training stage of image difference detection in this embodiment is similar to the image fusion process in the application stage, and will not be described again here.

[0122] Please see Figure 7 In some embodiments, step S104 may include steps S701 to S705.

[0123] Step S701: Take any one of the template image, the image to be detected, the fused image and the difference image as the input image and input it into the pre-trained tuple end-to-end learning model. First, perform average splitting on the input image to obtain the first image block after splitting.

[0124] Step S702: Set the vector dimension of the first image block to a preset vector dimension value, and then perform a straightening operation to process the two-dimensional image data into a one-dimensional sequence to obtain the second image block.

[0125] Step S703: Input the second image block into multiple preset Transformer modules for multiple information extraction and class pooling processes to obtain the third image block;

[0126] Step S704: By sharing feature extraction parameters, the third image blocks corresponding to the template image, the image to be detected, the fused image and the difference image are stacked respectively, and a linear layer is applied to compress them to the original size in terms of dimension, and then fused to obtain the fourth image block.

[0127] Step S705: Upsample the fourth image block to obtain a binary map with labeled difference regions.

[0128] For example, in the embodiments of this application, a multi-group end-to-end learning model based on convolutional self-attention can be pre-constructed. Specifically, in order to extract multi-scale high-dimensional features of images, a multi-group end-to-end learning model based on convolutional self-attention is constructed. This model is a multi-group end-to-end deep neural network. First, the template image, the image to be detected, the fused image, and the difference image are used as inputs to the autoencoder of the model, respectively. After passing through the encoder-decoder structure, the outputs are the restored template image I1, the image to be detected I2, the fused image O, and the difference image D. In the feature extraction network structure of the model, a hierarchical Transformer structure based on self-attention is adopted. The four inputs share the feature extraction network parameters, respectively extract high-dimensional feature maps, and stack them together to form the input of the decoder of the model. Then, after convolutional upsampling calculation, the difference region labeled binary map is obtained.

[0129] Specifically, in this embodiment, any one of the template image, the image to be detected, the fused image, and the difference image is used as the input image and fed into a pre-trained tuple end-to-end learning model. The input image is first averaged to obtain the first image patch. For example, in this embodiment, the encoder part of the model mainly consists of several modules based on a convolutional self-attention mechanism. Assuming the input image is of size H*W*3, after average splitting, an image patch of size H / 4*W / 4*48 is obtained, which is the first image patch.

[0130] Subsequently, in this embodiment, the vector dimension of the first image patch is set to a preset vector dimension value, and then a straightening operation is performed to process the two-dimensional image data into a one-dimensional sequence to obtain the second image patch. For example, the dimension of the feature vector of the input first image patch is changed to a preset vector dimension value C. The vector dimension value can be configured according to actual needs, and no specific restrictions are imposed here. The vector dimension value is a value that the Transformer module can accept. If it is set to 96, the vector size becomes H / 4*W / 4*96, and then after straightening, it becomes (H / 4*W / 4)*96. (H / 4*W / 4) is the length of the sequence, and 96 is the vector dimension. Thus, the two-dimensional image data is processed into a one-dimensional sequence to obtain the second image patch. In the obtained second image patch, the sequence with a length of 3136 is much larger than the length that ViTransformer can accept. Therefore, window-based self-attention is adopted. For example, each window has 7*7=49 small patches, so the sequence length becomes 49, which solves the problem of computational complexity.

[0131] Subsequently, in this embodiment, the second image block is input into multiple preset Transformer modules for multiple information extraction and pooling processes to obtain the third image block. For example, in this embodiment, there are multiple Transformer modules. The second image block is input into two Transformer modules based on a self-attention mechanism for information extraction, with an output size of H / 4*W / 4*C. To obtain feature information of multiple sizes, this embodiment also constructs a hierarchical Transformer, merging adjacent blocks to reduce the sequence length and increase the vector dimension, i.e., a pooling operation, with an output size of H / 8*W / 8*2C. Based on this, in this embodiment, multiple information extraction and pooling processes are performed, and then the image is input into two modules based on a self-attention mechanism for information extraction, with an output size still of H / 8*W / 8*2C. After a pooling operation, the output is H / 16*W / 16*4C, which is then connected to two Transformer modules based on a self-attention mechanism and a pooling layer, outputting a feature vector of H / 32*W / 32*8C, which is the third image block. It is understood that in the implementation of this application, different levels of Transformer modules can be set according to actual needs to convert the image to the required size. The above description is only based on one embodiment of this application and is not intended to limit the embodiments of this application.

[0132] It is understood that the feature fusion part of the model in this embodiment mainly fuses the features of the tuple. The four inputs are fed into the encoder branch, and by sharing the feature extraction parameters, four feature vectors of size H / 32*W / 32*8C are output respectively, which are the third image blocks. They are then stacked to form a feature vector of size H / 32*W / 32*32C. A linear layer is applied to compress the dimension to the original size 8C, and the fusion is used to obtain the fourth image block.

[0133] It is understood that the embodiments of this application use a model to process tuple image data, and tuples can effectively improve the performance of the model. During the image imaging process, due to the influence of factors such as acquisition equipment or lighting, a lot of background noise will inevitably be introduced. Therefore, background suppression is performed through some methods similar to attention. At the same time, the use of moving windows and restricting the use of self-attention within the window brings higher efficiency. By moving, interaction is made between two adjacent windows, that is, cross-window connection is made between upper and lower layers, thereby indirectly achieving a global modeling effect, and the computational complexity increases linearly with the image size.

[0134] Please see Figure 8 In some embodiments, step S705 may include steps S801 to S803:

[0135] Step S801: Input the fourth image block into the first linear layer, increase the feature dimension to twice the original dimension, and then use the rearrangement operation to expand the resolution of the input features to twice the input resolution and reduce the feature dimension to half the input dimension to obtain the fifth image block.

[0136] Step S802: Input the fifth image block into multiple preset Transformer modules and block extension layers for recovery processing to obtain the sixth image block;

[0137] Step S803: Input the sixth image block into the second linear layer to obtain a binary map of the difference region annotation.

[0138] For example, in this embodiment, the encoder section is designed to extract features and extract high-dimensional feature maps, which are then stacked together to form the input of the decoder. The decoder section corresponds to the encoder and is also a module based on a convolutional self-attention mechanism. Unlike the encoder, in this embodiment, a block expansion layer is used in the decoder to upsample the extracted depth features, reshaping the feature maps of adjacent dimensions into higher-resolution feature maps (e.g., 2× upsampling), and correspondingly halving the feature dimension.

[0139] Specifically, the fourth image patch is input into the first linear layer, increasing the feature dimension to twice the original dimension. Then, a rearrangement operation is used to double the resolution of the input features and reduce the feature dimension to half the input dimension, resulting in the fifth image patch. For example, a linear layer is added to the input fourth image patch, increasing the feature dimension to twice the original dimension. Then, a rearrangement operation is used to double the resolution of the input features and reduce the feature dimension to half the input dimension, i.e., H / 16*W / 16*4C, resulting in the fifth image patch.

[0140] Subsequently, in this embodiment, the fifth image block is input into a set of preset Transformer modules and a block expansion layer for restoration processing to obtain the sixth image block. The decoder first connects two Transformer modules based on a self-attention mechanism and a block expansion layer, outputting a feature map with a resolution and dimension of H / 8*W / 8*2C. Then, it connects two more Transformer modules based on a self-attention mechanism and a block expansion layer, outputting a feature map with a resolution and dimension of H / 4*W / 4*C. Finally, a block expansion layer is applied to restore the resolution to H*W*C / 4, thus achieving the restoration processing and obtaining the sixth image block. It can be understood that the Transformer module hierarchy in the decoder corresponds to the structure of the encoder to complete the image restoration process.

[0141] Finally, the sixth image block is input into the second linear layer to obtain a binary image of the difference region annotation. The first and second linear layers are only used to distinguish the relationship between the preceding and following parts. Specifically, in the embodiment of this application, the sixth image block is finally passed through a linear layer in the decoder to output an H*W*3 binary image of the difference region annotation.

[0142] To make the model more robust and reduce the spatial information loss caused by downsampling, the decoder part of this application adopts cross-layer connections to fuse multi-scale features from the encoder with upsampled features, connecting shallow and deep features together, followed by a linear layer. The size of the connected features is kept the same as the size of the upsampled features, and cross-layer connections are added at downsampling scales of 1 / 4, 1 / 8 and 1 / 16 respectively.

[0143] Please see Figure 9 In some embodiments, step S105 may include steps S901 to S904:

[0144] Step S901: Reduce the transparency of the fused image to obtain the third image;

[0145] Step S902: Reduce the transparency of the binary image labeled with the difference region to obtain the fourth image;

[0146] Step S903: Perform weighted processing on the third and fourth images to obtain the fused target difference image;

[0147] Step S904: Detect the highlighted parts in the target difference image and determine that the highlighted parts are the difference parts that display abnormalities, and obtain the image difference detection result.

[0148] For example, in the process of obtaining the image difference detection result in this application embodiment, it is also necessary to further process the binary map of the difference region annotation output by the model, including image fusion of the binary map of the difference region annotation.

[0149] For example, in this embodiment, the fused image and the binary map of the difference region annotation are fused. Since the model output is a binary difference map that does not contain element information, if it is desired to clearly see which element is different, image fusion can be used to highlight the difference. Specifically, this is achieved by reducing the transparency of the two images to be fused. Specifically, in this embodiment, the transparency of the fused image can be reduced to obtain a third image, and the transparency of the binary map of the difference region annotation can be reduced to obtain a fourth image. After the third and fourth images are fused, the other image will not be unclear. It can highlight some differences, such as element displacement, missing elements, and size consistency, while maintaining the color features. Then, the third and fourth images are weighted to obtain the fused target difference image.

[0150] It is understood that, in the default case of this application embodiment, the transparency of the binary image of the difference region annotation of the fused image and the model output is reduced by 50% at the same time, and a single image is synthesized, that is, a weighted summation with each weight of 0.5 to obtain the required target difference image. The transparency can also be adjusted according to the difference detection effect of the fused image, such as reducing the transparency of one image to above 50% and the other to below 50%.

[0151] For example, in this embodiment of the application, after fusing the binary images labeled with the difference regions, difference detection can be achieved, and it can be clearly seen which element is different. In this embodiment of the application, after the fusion is completed, the highlighted parts in the target difference image are detected, and the highlighted parts are the difference parts that show abnormalities, thus obtaining the image difference detection result.

[0152] In summary, to address the noise problem and reduce misjudgment of spurious difference information in this embodiment, the model utilizes difference images to retain sensitivity to spurious differences while effectively improving model performance using multivariate arrays. This is because the four inputs reflect the different features contained in the image pairs, generating less redundant information during computation. During imaging, background noise is inevitably introduced due to factors such as acquisition equipment or lighting. Therefore, background suppression is achieved through methods similar to attention. Simultaneously, the use of moving windows and limiting self-attention within the window results in higher efficiency. The movement enables interaction between adjacent windows, i.e., cross-window connections between upper and lower layers, thereby indirectly achieving a global modeling effect, and the computational complexity increases linearly with the image size.

[0153] Unlike methods in related technologies, the image difference detection method in this application learns image pseudo-differences and difference features in image pairs through four parallel streams. It exhibits strong learning ability for change regions at different scales, while also detecting pseudo-changes and resisting noise, effectively identifying and labeling differences of interest in image pairs. It can achieve intelligent end-to-end detection of image pair differences, with strong generalization and noise resistance, reducing labor costs and minimizing false positives and false negatives, showing promising application prospects.

[0154] For example, the image difference detection method in this application embodiment can be applied to any application field such as image difference detection, quality detection, and remote sensing image change detection, and no specific limitations are made here.

[0155] Please see Figure 10 This application also provides a model training method. Figure 10 This is an optional flowchart of the image difference detection method provided in the embodiments of this application. Figure 10The method may include, but is not limited to, steps S1001 to S1006.

[0156] Step S1001: Obtain the template sample image and the sample image to be detected;

[0157] Step S1002: The template sample image and the sample image to be detected are fused to obtain the fused sample image;

[0158] Step S1003: Perform post-difference calculation on the template sample image and the sample image to be detected to obtain the difference sample image;

[0159] Step S1004: Perform mask processing on the image to be detected, and mark the different regions between the image to be detected and the template image to obtain a binarized difference annotation map.

[0160] Step S1005: Input the template sample image, the sample image to be detected, the fused sample image and the difference sample image into the multivariate end-to-end learning model respectively to obtain the binary sample image of the difference region annotation;

[0161] Step S1006: Calculate the total loss value of the model training based on the difference annotation map and the difference region annotation binary sample map, and adjust the parameters of the multivariate end-to-end learning model according to the total loss value to obtain the trained multivariate end-to-end learning model.

[0162] For example, in the embodiments of this application, a tuple-based end-to-end learning model can be pre-trained. During the training process, the embodiments of this application can acquire multiple sample data, including template sample images and sample images to be detected. The model training method can also be applied to the image difference detection system described in the above embodiments, and no specific limitations are imposed here.

[0163] The template sample image is an image used as a standard template, similar to the template image in the above application process embodiment. The sample image to be detected is a target image used for image difference detection during training, similar to the template image in the above application process embodiment. In this application embodiment, by acquiring the template sample image and the sample image to be detected, these two images can be processed to detect whether there is a difference between the sample image to be detected and the template sample image, thereby finally completing the training of the model.

[0164] It is understood that in the embodiments of this application, multiple template sample images or multiple sample images to be detected can be obtained. The template sample images and sample images to be detected can be in a one-to-one correspondence or a one-to-many correspondence, which will not be elaborated here.

[0165] In this embodiment, by acquiring a template image and an image to be detected, image pairs can be established, which can then be used to detect the differences between the image pairs. As training samples, the sample pairs consisting of the template sample image and the image to be detected can contain multiple differing image pairs and indistinguishable image pairs, so that the model can be trained for different difference situations.

[0166] For example, in this embodiment of the application, image fusion is required between the template sample image and the sample image to be detected to obtain a fused sample image, which can then be used for training subsequent difference detection. It should be noted that the fused sample image can also be understood as a synthetic image. Similar to the fused image in the above application process, the fused sample image can highlight some differences while maintaining the color features after image processing, such as highlighting element displacement, missing elements, size consistency, etc. between the two images before fusion.

[0167] For example, in this embodiment, post-difference calculation is performed on the template sample image and the sample image to be detected to obtain a difference sample image, which can then be used for training subsequent difference detection. It should be noted that the difference sample image is obtained after post-difference calculation between the two images. Similar to the difference image in the above application process, it contains important key information for difference detection, especially elemental differences. Because the direct subtraction operation causes the difference image to lose the image details contained in the original image, this embodiment utilizes the difference image in subsequent models to retain sensitivity to false differences, solve noise problems, and reduce misjudgments of false difference information.

[0168] For example, in order to train the model, it is necessary to establish a comparison between the model's output results to verify whether the model's output results are accurate. Therefore, in this embodiment, the image to be detected is masked to mark the different regions between the image to be detected and the template image, thus obtaining a binarized difference annotation map.

[0169] Specifically, the difference annotation map established in this embodiment is a binary segmentation map. Specifically, in the masking process, this embodiment uses black pixels as the base, with the same size as the template image, and uses a white mask to mark the different regions between the sample image to be detected and the template sample image, that is, a 0 / 1 binary segmentation map, thereby obtaining the difference annotation map.

[0170] In this embodiment, the data input to the multivariate end-to-end learning model for training includes four elements: a template sample image, a sample image to be detected, a fused sample image, and a difference sample image, forming a multivariate image set. Inputting four images simultaneously effectively improves the model's performance. The multivariate end-to-end learning model can be trained using multiple sample data sets, thereby improving its ability to process input multivariate data after training and outputting more accurate binary maps with labeled difference regions.

[0171] It should be noted that, as described above, the template image, the image to be detected, the fused image, and the difference image in this application embodiment reflect different image features, thus generating less redundant information during the calculation process. For example, the model in this application embodiment can use a self-attention mechanism layer based on convolution-like structures to construct hierarchical feature representations through small image patches and layer-by-layer neighborhood merging, enabling the extraction of visual features at different levels, thereby improving the accuracy of anomaly detection. No specific limitations are imposed here.

[0172] During training, the multivariate end-to-end learning model processes the input sample data and outputs a binary sample map with labeled difference regions. This binary map determines which elements in the image are different. During training, the total training loss is calculated based on the labeled difference map and the binary sample map, and the parameters of the multivariate end-to-end learning model are adjusted accordingly to obtain the trained multivariate end-to-end learning model.

[0173] It is understood that during the training process, the template sample image, the sample image to be detected, the fused sample image and the difference sample image can be obtained directly. That is, the training process may not perform image fusion and post-difference calculation processing on the template sample image and the sample image to be detected in the system, but directly obtain four sample images and establish tuple data. No specific restrictions are made here.

[0174] Please see Figure 11 This application also provides an image difference detection system that can implement the above-described image difference detection method. The image difference detection system includes:

[0175] Image sampling module 1101 is used to acquire template image and image to be detected;

[0176] The fusion module 1102 is used to fuse the template image and the image to be detected to obtain a fused image;

[0177] The difference module 1103 is used to perform post-difference calculation on the template image and the image to be detected to obtain a difference image;

[0178] The model processing module 1104 is used to input the template image, the image to be detected, the fused image and the difference image into the pre-trained multi-group end-to-end learning model to obtain a binary map of the difference region annotation.

[0179] The difference determination module 1105 is used to determine the difference parts that display anomalies from the binary map labeled with difference regions, and obtain the image difference detection result.

[0180] For example, the image difference detection system (hereinafter referred to as the system) can execute the image difference detection method in the above embodiments. In the process of image difference detection, the system needs to acquire a template image and an image to be detected.

[0181] The template image is an image used as a standard template, also called a template image. The image to be detected is a target image used for image difference detection. In this embodiment, by acquiring the template image and the image to be detected, these two images can be processed to detect whether there are differences between the image to be detected and the template image, thereby finally completing the image difference detection.

[0182] It is understood that in the embodiments of this application, multiple template images or multiple images to be detected can be obtained. There can be a one-to-one correspondence between the template images and the images to be detected, that is, one template image corresponds to one image to be detected. In addition, there can be a one-to-many correspondence between the template images and the images to be detected, that is, one template image corresponds to multiple images to be detected. This embodiment of the application does not impose specific limitations on this.

[0183] In this embodiment of the application, by acquiring a template image and an image to be detected, an image pair can be established, which can then be used to detect the differences between the image pairs.

[0184] For example, in this embodiment of the application, image fusion is required between the template image and the image to be detected to obtain a fused image, which can then be used for subsequent difference detection. It should be noted that the fused image can also be understood as a synthetic image. The fused image can highlight some differences while maintaining the color features after image processing, such as highlighting element displacement, missing elements, size consistency, etc. between the two images before fusion.

[0185] For example, in this embodiment, post-difference calculation is performed on the template image and the image to be detected to obtain a difference image, which can then be used for subsequent difference detection. It should be noted that the difference image is obtained after post-difference calculation between the two images and contains important key information for difference detection, especially elemental differences. Because direct subtraction causes the difference image to lose image details contained in the original image, this embodiment utilizes the difference image in subsequent models to retain sensitivity to false differences, solve noise problems, and reduce misjudgments of false difference information.

[0186] For example, in this embodiment of the application, a multi-tuple end-to-end learning model (hereinafter referred to as the model) can be pre-trained. This model is a neural network model that can be pre-set in the system and used for image difference detection. In this embodiment of the application, the data input to the multi-tuple end-to-end learning model includes four parts: a template image, the image to be detected, a fused image, and a difference image, which together form a multi-tuple image. Inputting four images simultaneously can effectively improve the performance of the model.

[0187] As is understandable, a tuple-based end-to-end learning model is an end-to-end model; it takes image data as input and outputs an image. A tuple-based end-to-end learning model can be trained on multiple sample datasets, thereby improving its ability to process input tuple data after training and outputting more accurate binary maps with labeled difference regions.

[0188] It should be noted that, as described above, the template image, the image to be detected, the fused image, and the difference image in this application embodiment reflect different image features, thus generating less redundant information during the calculation process. For example, the model in this application embodiment can use a self-attention mechanism layer based on convolution-like structures to construct hierarchical feature representations through small image patches and layer-by-layer neighborhood merging, enabling the extraction of visual features at different levels, thereby improving the accuracy of anomaly detection. No specific limitations are imposed here.

[0189] After processing the input data, the tuple-based end-to-end learning model can output a binary map with labeled difference regions. This binary map is used to determine which elements in the image are different.

[0190] For example, in the embodiments of this application, after obtaining the binary map of the difference region annotation output by the model, since the binary map can determine which element in the image is different, the difference part that shows anomalies can be determined from the binary map of the difference region annotation. This difference part is the part that is different between the image to be detected and the template image, and thus the image difference detection result can be obtained based on the difference part.

[0191] It is understandable that image difference detection results can characterize and indicate the differences between the image to be detected and the template image. After obtaining the image difference detection results, the user can understand the differences between the images. Alternatively, the image difference detection results can be sent to other terminal devices for parsing. After receiving the image difference detection results, the terminal devices can parse the results to obtain the differences between the image to be detected and the template image.

[0192] This application embodiment learns image pseudo-differences and difference features in image pairs through four parallel streams, exhibiting strong learning ability for regions of change at different scales. It can realize intelligent end-to-end detection of image pair differences, with strong generalization ability and noise resistance, which can reduce labor costs, reduce false detections and false negatives, and improve the accuracy of image difference detection.

[0193] The specific implementation of this image difference detection system is basically the same as the specific embodiments of the image difference detection method described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this application, the image difference detection system may also be equipped with other functional modules to implement the image difference detection method described above.

[0194] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described image difference detection method or model training method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0195] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0196] The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0197] The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to execute the image difference detection method or model training method of the embodiments of this application.

[0198] The input / output interface 1203 is used to implement information input and output;

[0199] The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0200] Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204);

[0201] The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0202] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image difference detection method or model training method.

[0203] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0204] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0205] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0208] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0209] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0210] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0211] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0213] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An image difference detection method, characterized in that, The method includes: Obtain the template image and the image to be detected; The template image and the image to be detected are fused together to obtain a fused image. The template image and the image to be detected are subjected to post-difference calculation to obtain a difference image; The template image, the image to be detected, the fused image, and the difference image are respectively input into a pre-trained tuple end-to-end learning model to obtain a binary image of the difference region annotation; The image difference detection result is obtained by identifying the abnormal difference portion from the binary image labeled with the difference region. Before performing image fusion between the template image and the image to be detected, the method further includes: Feature detection is performed on the template image and the image to be detected, and feature points between the template image and the image to be detected are matched; Calculate the feature point similarity between the template image and the image to be detected, sort them by similarity to obtain a ranking result, and determine multiple target feature points from the ranking result; Affine transformation is performed on the image to be detected based on the target feature points to obtain the image to be detected after image registration. The image to be detected after image registration is used to perform image fusion and post-difference calculation with the template image. The step of inputting the template image, the image to be detected, the fused image, and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary map of the difference region annotation includes: The template image, the image to be detected, the fused image, and the difference image are used as input images and input into a pre-trained tuple end-to-end learning model. The input image is first split into an average value to obtain the first image block after splitting. The vector dimension of the first image block is set to a preset vector dimension value, and then a straightening operation is performed to process the two-dimensional image data into a one-dimensional sequence to obtain the second image block. The second image block is input into multiple preset Transformer modules for multiple information extraction and class pooling processes to obtain the third image block; By sharing feature extraction parameters, the third image blocks corresponding to the template image, the image to be detected, the fused image and the difference image are stacked respectively, and a linear layer is applied to compress them to the original size in terms of dimension, and then fused to obtain a fourth image block; The fourth image block is upsampled to obtain a binary map with labeled difference regions.

2. The image difference detection method according to claim 1, characterized in that, The tuple-based end-to-end learning model is trained through the following steps: Obtain the template sample image and the sample image to be detected; The template sample image and the sample image to be detected are fused together to obtain a fused sample image. The template sample image and the sample image to be detected are subjected to post-difference calculation to obtain the difference sample image; The image to be detected is masked to mark the different regions between the image to be detected and the template image, resulting in a binarized difference annotation map. The template sample image, the sample image to be detected, the fused sample image, and the difference sample image are respectively input into the multivariate end-to-end learning model to obtain a binary sample image with difference region annotation; The total loss value for model training is calculated based on the difference-annotated map and the difference region labeled binary sample map, and the parameters of the tuple end-to-end learning model are adjusted based on the total loss value to obtain the trained tuple end-to-end learning model.

3. The image difference detection method according to claim 2, characterized in that, The step of calculating the total loss value for model training based on the difference-annotated map and the difference-region-annotated binary sample map includes: Based on the differences between the template sample image and the sample image to be detected, training sample pairs are established. Based on each training sample pair, the contrast loss is calculated using the corresponding difference annotation map and the difference region annotation binary sample map to obtain a first loss value. Calculate the distance between the difference-annotated map and the difference-region-annotated binary sample map to obtain the second loss value; The total loss value for model training is obtained by weighting the first loss value and the second loss value.

4. The image difference detection method according to claim 3, characterized in that, The step of performing a comparison loss calculation on the corresponding difference-annotated map and the difference-region-annotated binary sample map based on each of the training samples to obtain a first loss value includes: Obtain the number of training sample pairs to get the number of sample pairs; The differences between the template sample image and the sample image to be detected in each training sample pair are obtained to obtain sample labels that represent samples that are the same or matched; Obtain the Euclidean distance between the difference-annotated map and the difference-region-annotated binary sample map corresponding to each training sample pair, and obtain the sample Euclidean distance; Based on the number of sample pairs, the sample labels, and the Euclidean distance between the samples, a contrast loss is calculated for the difference-annotated map and the difference-region-annotated binary sample map corresponding to each training sample pair to obtain a first loss value.

5. The image difference detection method according to claim 1, characterized in that, The step of fusing the template image with the image to be detected to obtain a fused image includes: Reduce the transparency of the template image to obtain the first image; The transparency of the image to be detected is reduced to obtain a second image; The first image and the second image are weighted to obtain the fused image.

6. The image difference detection method according to claim 1, characterized in that, The upsampling of the fourth image block to obtain a binary map with labeled difference regions includes: The fourth image block is input into the first linear layer, increasing the feature dimension to twice the original dimension. Then, a rearrangement operation is used to expand the resolution of the input features to twice the input resolution and reduce the feature dimension to half the input dimension, resulting in the fifth image block. The fifth image block is input into multiple preset Transformer modules and a block extension layer for recovery processing to obtain the sixth image block; The sixth image block is input into the second linear layer to obtain a binary map of the difference region annotation.

7. The image difference detection method according to claim 1, characterized in that, The step of determining the abnormal difference portion from the labeled binary image of the difference region to obtain the image difference detection result includes: Reduce the transparency of the fused image to obtain a third image; Reduce the transparency of the binary image labeled with the difference region to obtain the fourth image; The third image and the fourth image are weighted to obtain a fused target difference image. The highlighted parts in the target difference image are detected, and the highlighted parts are identified as difference parts with display abnormalities, thus obtaining the image difference detection result.

8. A model training method, characterized in that, The method includes: Obtain the template sample image and the sample image to be detected; The template sample image and the sample image to be detected are fused together to obtain a fused sample image. The template sample image and the sample image to be detected are subjected to post-difference calculation to obtain the difference sample image; The image to be detected is masked to mark the different regions between the image to be detected and the template image, resulting in a binarized difference annotation map. The template sample image, the sample image to be detected, the fused sample image, and the difference sample image are respectively input into the multivariate end-to-end learning model to obtain a binary sample image with difference region annotation; The total loss value for model training is calculated based on the difference-annotated map and the difference region labeled binary sample map, and the parameters of the multivariate end-to-end learning model are adjusted based on the total loss value to obtain the trained multivariate end-to-end learning model. In the model training phase of image difference detection, after obtaining the sample image to be detected, image registration of the template sample image is required. The tuple end-to-end learning model is used to execute the image difference detection method as described in claim 1.

9. An image difference detection system, characterized in that, The system includes: The image sampling module is used to acquire the template image and the image to be detected; The fusion module is used to fuse the template image and the image to be detected to obtain a fused image; The difference module is used to perform post-difference calculation on the template image and the image to be detected to obtain a difference image; The model processing module is used to input the template image, the image to be detected, the fused image and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary image of the difference region annotation. The difference determination module is used to determine the difference portion that displays anomalies from the binary image labeled with the difference region, and to obtain the image difference detection result; Before fusing the template image with the image to be detected, the method further includes: Feature detection is performed on the template image and the image to be detected, and feature points between the template image and the image to be detected are matched; Calculate the feature point similarity between the template image and the image to be detected, sort them by similarity to obtain a ranking result, and determine multiple target feature points from the ranking result; Affine transformation is performed on the image to be detected based on the target feature points to obtain the image to be detected after image registration. The image to be detected after image registration is used to perform image fusion and post-difference calculation with the template image. The step of inputting the template image, the image to be detected, the fused image, and the difference image into a pre-trained tuple end-to-end learning model to obtain a binary map of the difference region annotation includes: The template image, the image to be detected, the fused image, and the difference image are used as input images and input into a pre-trained tuple end-to-end learning model. The input image is first split into an average value to obtain the first image block after splitting. The vector dimension of the first image block is set to a preset vector dimension value, and then a straightening operation is performed to process the two-dimensional image data into a one-dimensional sequence to obtain the second image block. The second image block is input into multiple preset Transformer modules for multiple information extraction and class pooling processes to obtain the third image block; By sharing feature extraction parameters, the third image blocks corresponding to the template image, the image to be detected, the fused image and the difference image are stacked respectively, and a linear layer is applied to compress them to the original size in terms of dimension, and then fused to obtain a fourth image block; The fourth image block is upsampled to obtain a binary map with labeled difference regions.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the image difference detection method according to any one of claims 1 to 7 and the model training method according to claim 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image difference detection method according to any one of claims 1 to 7 and the model training method according to claim 8.

Images