Image change recognition method, device, equipment and storage medium

Abstract

The present application provides an image change recognition method, device, equipment and storage medium, which relates to the field of artificial intelligence technology and can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: obtaining a group of images to be recognized, the group of images to be recognized includes a reference image and a corresponding comparison image, and performing feature extraction on the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image; then performing difference information extraction on the first image feature and the second image feature to obtain at least one difference feature, and performing feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain a target fusion feature; and then performing change recognition on the target fusion feature based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image. The present application can improve the universality and deployability of the image change recognition method.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image change recognition method, device, equipment and storage medium. Background Art

[0002] Image change recognition is an important computer vision task that aims to identify the changed parts in an image group. It can be widely used in various scenarios such as industrial quality inspection, map applications, disaster assessment, medical image processing, agricultural problem detection, and security identification. Existing image change recognition methods usually use a reconstructed change detection network to perform overall change recognition on the image group. The design is difficult, and its network structure is complex, making it difficult to deploy in industrial applications. It also requires a large amount of specialized change recognition training data for training. The scarcity of change recognition samples makes it difficult to be effectively trained, resulting in poor change recognition performance. Summary of the invention

[0003] The present application provides an image change recognition method, apparatus, device and storage medium, which specifically include the following contents.

[0004] In one aspect, the present application provides an image change recognition method, the method comprising:

[0005] Acquire a group of images to be identified, wherein the group of images to be identified includes a reference image and a corresponding comparison image;

[0006] Extracting features from the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image;

[0007] Extracting difference information between the first image feature and the second image feature to obtain at least one difference feature;

[0008] Performing feature fusion processing on the first image feature, the second image feature and the at least one difference feature to obtain a target fusion feature;

[0009] Based on a preset semantic segmentation algorithm, changes are identified on the target fusion features to obtain a change identification result between the reference image and the comparison image.

[0010] Another aspect provides an image change recognition device, the device comprising:

[0011] An image group acquisition module is used to acquire an image group to be identified, wherein the image group to be identified includes a reference image and a corresponding comparison image;

[0012] Feature extraction module: used to extract features from the reference image and the comparison image respectively, to obtain a first image feature of the reference image and a second image feature of the comparison image;

[0013] A difference feature extraction module: used for extracting difference information between the first image feature and the second image feature to obtain at least one difference feature;

[0014] A time domain feature fusion network: used for performing feature fusion processing on the first image feature, the second image feature and the at least one difference feature to obtain a target fusion feature;

[0015] Change recognition module: used to identify changes in the target fusion features based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image.

[0016] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the image change recognition method as described above.

[0018] On the other hand, an image change recognition terminal is provided, the terminal includes a processor and a memory, the device includes a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method as described above.

[0019] On the other hand, a server is provided, the server includes a processor and a memory, the device includes a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method as described above.

[0020] On the other hand, a computer program product or a computer program is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the image change recognition method as described above is implemented.

[0021] The image change recognition method, device, equipment, storage medium, terminal, server and computer program product provided by the present application have the following technical effects:

[0022] The technical solution of the present application first obtains a group of images to be identified, which includes a reference image and a corresponding comparison image, and extracts features from the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image; then extracts difference information from the first image feature and the second image feature to obtain at least one difference feature, and performs feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain a target fusion feature; and then performs change recognition on the target fusion feature based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image. In this way, the difference information extraction between the reference image and the comparison image is decoupled from feature fusion and feature recognition, and then network modules can be designed separately for difference information extraction, feature fusion and feature recognition, thereby improving the universality of the image change recognition method and realizing the migration of the semantic segmentation method in the change recognition task. The existing semantic segmentation network or a pre-trained semantic segmentation network with a small amount of modification can be used to realize change recognition, reduce the model design and structural complexity, improve deployability, and reduce training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0025] Figure 2 It is a flowchart of an image change recognition method provided by an embodiment of the present application;

[0026] Figure 3 is a flowchart of another image change recognition method provided by an embodiment of the present application;

[0027] Figure 4 is a flowchart of another image change recognition method provided by an embodiment of the present application;

[0028] Figure 5 is a flowchart of another image change recognition method provided by an embodiment of the present application;

[0029] Figure 6It is a schematic diagram of a framework of a target recognition model provided in an embodiment of the present application;

[0030] Figure 7 is a flowchart of another image change recognition method provided by an embodiment of the present application;

[0031] Figure 8 It is a network framework diagram of a target recognition model provided by an embodiment of the present application;

[0032] Fig. 9 is a group of images to be identified provided by an embodiment of the present application;

[0033] Fig.10 It is a network framework diagram of a time domain feature fusion network provided in an embodiment of the present application;

[0034] Fig.11 It is a network framework diagram of a spatial domain feature fusion network provided in an embodiment of the present application;

[0035] Fig.12 It is a visualization diagram of the output of each network module in the target recognition model provided in the embodiment of the present application;

[0036] Fig.13 is a group of images to be identified provided by an embodiment of the present application;

[0037] Fig.14 is another group of images to be identified provided by an embodiment of the present application;

[0038] Fig.15 It is a schematic diagram of a framework of an image change recognition device provided in an embodiment of the present application;

[0039] Fig.16 It is a hardware structure block diagram of an electronic device for an image change recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0042] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0043] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image change recognition, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0044] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. This application mainly involves technologies such as image change recognition, semantic understanding, machine learning and deep learning, which are specifically explained through the following embodiments.

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0048] Semantic segmentation: Semantic segmentation refers to classification at the pixel level, where pixels belonging to the same class are grouped into one class.

[0049] Backbone network: mainly refers to the network for feature extraction.

[0050] ReLU (Rectified Linear Unit): is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0051] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, the application environment may include at least a server 01 and a terminal 02. In actual applications, the server 01 and the terminal 02 may be directly or indirectly connected via wired or wireless communication to achieve interaction between the terminal 02 and the server 01, and this application does not limit this.

[0052] In the embodiment of the present application, server 01 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Specifically, the server may include physical devices, which may specifically include network communication submodules, processors, and memories, etc., and may also include software running in physical devices, which may specifically include applications, etc.

[0053] Specifically, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". Among them, artificial intelligence cloud service is generally also called AIaaS (AI as a Service, Chinese for "AI as a Service"). This is a mainstream service mode of artificial intelligence platform. Specifically, AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through API interfaces. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0054] In the embodiment of the present application, the terminal 02 may include a smart phone, a desktop computer, a tablet computer, a laptop computer, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, an intelligent voice interaction device, a smart home appliance, an intelligent wearable device, a vehicle-mounted terminal device, an aircraft and other types of physical devices, and may also include software running in the physical device, such as an application program. Specifically, the server 01 may be used to provide an image change recognition service for the image group to be identified, and obtain a change recognition result, which may specifically include performing feature extraction, difference information extraction, feature fusion and change recognition processing on the image group to be identified based on the target recognition model to obtain a change recognition result. The server 01 may also be used to provide a training service for the target recognition model, which may specifically include performing model training on the initial recognition model based on the sample image group in the training sample set and the corresponding sample label to obtain a target recognition model. The terminal 02 may be used to send the image group to be identified to the server 01, or may run the above-mentioned target recognition model, and provide an image change recognition service.

[0055] Furthermore, it is understandable that Figure 1 What is shown is only an application environment of an image change recognition method, and the application environment may include more or fewer nodes, and the present application does not impose any limitation thereto.

[0056] The application environment involved in the embodiment of the present application, or the server 01 and the terminal 02 in the application environment, can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned image change recognition service, model training service and data storage service, etc.

[0057] The following is an introduction to an image change recognition method of the present application based on the above application environment. The present application embodiment can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2 It is a flowchart of an image change recognition method provided in an embodiment of the present application. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include the following steps S201-S209.

[0058] S201: Obtain a group of images to be identified.

[0059] Specifically, the image group to be identified includes a reference image and a corresponding comparison image. The comparison image refers to an image that has local differences from the reference image. For example, the reference image and the comparison image are images of the same object at different times. For example, the reference image is a street view image at an earlier time, and the comparison image is a street view image of the same area at a later time. Fig.13 , Fig.13 The upper left image is the reference image, and the upper right image is the comparison image; or, the reference image and the comparison image are images of different objects of the same type, please refer to Fig.14 , Fig.14 The left middle image is the reference image, and the right middle image is the comparison image.

[0060] S203: Perform feature extraction on the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image.

[0061] In an embodiment of the present application, a reference image and a comparison image can be input into a feature extraction network for feature extraction, wherein the first image feature represents a feature in the reference image, and the second image feature represents a feature in the comparison image, and the image feature may include a feature map of at least one feature scale. The feature scales and corresponding feature extraction methods of the first image feature and the second image feature are the same. Specifically, feature extraction can be performed based on a backbone network, and the backbone network can be constructed based on a preset semantic segmentation algorithm, which is a semantic segmentation algorithm implemented based on a neural network.

[0062] S205: Extract difference information from the first image feature and the second image feature to obtain at least one difference feature.

[0063] Specifically, the types of difference features may include but are not limited to disappearing features, newly added features, exchange features, and overall transformation features. The disappearing features represent the features of objects that exist in the reference image but do not exist in the comparison image, the newly added features represent the features of objects that do not exist in the reference image but exist in the comparison image, the exchange features represent the features of objects that exist in the reference image but disappear in the comparison image and are replaced by other objects, and the overall transformation features represent the overall difference between the features of the reference image and the comparison image; accordingly, the difference information extraction is to extract at least one of the disappearing features, newly added features, exchange features, and overall transformation features between the first image feature and the second image feature. The process of difference information extraction may include: performing subtraction processing on the first image feature and the second image feature, and then mapping and convolution processing on the features obtained by the subtraction processing to obtain the difference features. The subtraction processing here may be to subtract the second image feature from the first image feature, to subtract the first image feature from the second image feature, or to perform feature difference processing on the first image feature and the second image feature to obtain the absolute value of the subtraction between the two. The mapping processing here is to map each pixel in the feature obtained by the subtraction processing based on a preset mapping method, for example, mapping based on a preset linear rectification method, such as the ReLU method.

[0064] In the case where at least one of the differentiating features includes a disappearing feature, please refer to Figure 3 , S205 includes the following steps S2051-S2053.

[0065] S2051: Subtract the second image feature from the first image feature to obtain a first distinguishing feature.

[0066] S2052: Perform feature mapping processing on the first distinguishing feature based on a preset mapping method to obtain a first intermediate feature.

[0067] S2053: Perform convolution processing on the first intermediate feature to obtain a disappearing feature.

[0068] Specifically, the second image feature is subtracted from the first image feature, and each pixel in the obtained first distinguishing feature is mapped to obtain a first intermediate feature, and after convolution processing to extract information, a disappearing feature is obtained. In an example, f disappear =ReLU(f 0 -f 1 ), where f disappear is the first intermediate feature, ReLU is the preset mapping algorithm used, and f 0 is the first image feature, f 1 is the second image feature.

[0069] In the case where at least one of the differentiating features includes a new feature, please refer to Figure 4, S205 includes the following steps S2054-S2056.

[0070] S2054: Subtract the first image feature from the second image feature to obtain a second distinguishing feature.

[0071] S2055: Perform feature mapping processing on the second distinguishing feature based on a preset mapping method to obtain a second intermediate feature.

[0072] S2056: Perform convolution processing on the second intermediate feature to obtain a new feature.

[0073] Specifically, the first image feature is subtracted from the second image feature, and each pixel in the obtained second distinguishing feature is mapped to obtain a second intermediate feature, and after convolution processing to extract information, a new feature is obtained. In an example, f appear =ReLU(f 1 -f 0 ), where f disappear It is the second intermediate feature.

[0074] In the case where at least one difference feature includes an overall transformation feature, refer to Figure 5 , S205 includes the following steps S2057 and S2058.

[0075] S2057: Perform feature difference processing on the first image feature and the second image feature to obtain a third intermediate feature.

[0076] S2058: Perform convolution processing on the third intermediate feature to obtain an overall transformation feature.

[0077] Specifically, the feature difference processing here can be implemented in the following manner: the first intermediate feature and the second intermediate feature are obtained respectively by the aforementioned S2051-S2052 and S2054-S2055, and the first intermediate feature and the second intermediate feature are added to obtain the third intermediate feature. The corresponding expression is f different =ReLU(f 1 -f 0 )+ReLU(f 0 -f 1 ), where f different is the third intermediate feature. Alternatively, take the absolute value of the first image feature and the second image feature to obtain the third intermediate feature. The corresponding expression is f different =max(f 0 ,f 0 )-min(f 0 ,f 0 ), and then convolution is performed on the third intermediate feature to obtain the overall transformation feature.

[0078] It should be noted that the convolution processing in S2053, S2056 and S2058 may adopt the same convolution method or different convolution methods.

[0079] S207: Perform feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain a target fusion feature.

[0080] Specifically, the fusion processing here may include but is not limited to splicing, adding or linearly combining the first image feature, the second image feature and at least one difference feature, such as simple splicing or simple addition. It should be noted that if the first image feature, the second image feature and the difference feature have multiple feature scales of the same number of scale types, the feature fusion processing refers to feature fusion of feature maps of the same feature scale in the first image feature, the second image feature and the difference feature to obtain a target fusion feature of each feature scale.

[0081] Specifically, when the difference features include disappearing features, newly added features, and overall transformed features, the target fusion feature fuses these changes.

[0082] S209: Perform change recognition on the target fusion features based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image.

[0083] Specifically, feature fusion is performed based on a preset semantic segmentation algorithm. The preset semantic segmentation algorithm here and the preset semantic segmentation algorithm for feature extraction mentioned above may belong to the algorithm adopted by the same semantic segmentation network. After obtaining the target fusion feature, the change recognition network is used to perform change recognition on the target fusion feature to obtain a change recognition result, wherein the change recognition network may be a semantic segmentation network based on the preset semantic segmentation algorithm. The change recognition result represents the difference and difference category between the reference image and the comparison image, including the predicted label of each pixel in the comparison image. For example, the output change recognition result is a binary image, 1 represents the recognized difference pixel, and 0 represents the recognized pixel without difference. Please refer to Fig.13 , Fig.13 The lower right part of the middle figure shows the visualized change recognition result. Specifically, in the case of multi-scale features, the target fusion features can be convolved through the spatial domain feature fusion network to unify the scale and obtain the target recognition features for change recognition. Then, the target recognition features are changed based on the preset semantic segmentation algorithm to obtain the change recognition result.

[0084] In summary, the difference information extraction between the reference image and the comparison image is decoupled from feature fusion and feature recognition, and then network modules can be designed respectively for difference information extraction, feature fusion and feature recognition, thereby improving the universality of the image change recognition method and realizing the migration of the semantic segmentation method in the change recognition task. The existing semantic segmentation network or a pre-trained semantic segmentation network with a small amount of modification can be used to achieve change recognition, reducing the complexity of model design and structure, improving deployability and reducing training costs.

[0085] In some embodiments, S203 specifically includes: using the same backbone network to perform multi-scale feature extraction on the reference image and the comparison image respectively to obtain the first image feature and the second image feature.

[0086] Specifically, the same backbone network is used to extract features of the reference image and the comparison image, that is, the same feature extraction method is used to extract features of the two images, which can ensure high comparability between the first image features and the second image features, remove interference features, and optimize feature comparison results.

[0087] Specifically, the first image feature and the second image feature respectively include image features of a preset number of feature scales. The preset number can be set based on actual needs. For example, the preset number is 4, and the feature scales are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input scale, respectively. In this way, through multi-scale feature extraction, multi-scale features can be combined to perform difference feature extraction and feature fusion, thereby improving the accuracy of feature extraction and difference feature extraction, and thereby improving the accuracy of change recognition results.

[0088] Accordingly, S205 may specifically include: for each feature scale, extracting difference information from the first image feature and the second image feature to obtain at least one difference feature corresponding to each feature scale. S207 may specifically include: for each feature scale, performing feature fusion processing on the first image feature, the second image feature and the at least one difference feature to obtain a target fusion feature corresponding to each feature scale.

[0089] Specifically, in the case where the first image feature and the second image feature include feature maps of multiple scales, the aforementioned difference information extraction process needs to perform difference feature extraction for each feature scale, and then obtain at least one difference feature at each feature scale. If the difference feature types include disappearing features, newly added features, exchange features, and overall transformation features, then the disappearing features, newly added features, exchange features, and overall transformation features at each feature scale are extracted. For example, when performing difference information extraction for the first image feature and the second image feature at a 1 / 4 scale, at least one difference feature at a 1 / 4 scale is obtained. Similarly, at least one difference feature at scales of 1 / 8, 1 / 16, and 1 / 32 is obtained.

[0090] Further, in the case of multi-scale feature extraction, S209 may include the following steps S2091 and S2092.

[0091] S2091: Performing scale fusion processing on the target fusion features corresponding to each feature scale to obtain target recognition features.

[0092] S2092: Perform change recognition on the target recognition features based on a preset semantic segmentation algorithm to obtain a change recognition result.

[0093] Specifically, after obtaining the target fusion features at multiple feature scales, the above-mentioned scale fusion is realized based on upsampling and convolution. In some embodiments, a separate scale fusion process can be performed for each category difference feature at each feature scale to obtain a preset number of intermediate scale features of the same scale, and then the intermediate scale features are spliced ​​to obtain the target recognition feature. In other embodiments, convolution and upsampling processes can be performed on each target fusion feature of a preset number of feature scales except the maximum scale to obtain the first intermediate recognition feature corresponding to each target fusion feature of other scales, and then the first intermediate recognition feature of the smaller feature scale is fused with the adjacent intermediate recognition feature of the larger scale to obtain the second intermediate recognition feature of the larger scale, and each second intermediate recognition feature is further scale fused and convoluted to obtain the target recognition feature of the input scale. Specifically, before the scale fusion of each second intermediate recognition feature is performed, it can also be upsampled to unify the feature scale of each second intermediate recognition feature.

[0094] In summary, multi-scale features are combined to perform multi-category difference feature extraction, fusion and change recognition to optimize the accuracy of image change recognition.

[0095] In the embodiment of the present application, the image change recognition method of the present application is implemented based on the target recognition model. For details, please refer to Figure 6, the target recognition model includes a backbone network, a temporal feature fusion network (Merge Temporal Features, MTF), a spatial feature fusion network (Merge Spatial Features, MSF) and a feature recognition network. Among them, the backbone network is used to extract features from the reference image and the comparison image, and can include a pre-trained feature extraction network, such as a backbone network based on a semantic segmentation model. The pre-trained feature extraction network may include but is not limited to VGG (Visual Geometry Group), ResNet (Deep Residual Network), MobileNet and Swin Transformer, etc., or other deep neural networks. The temporal feature fusion network is used to extract difference information and perform feature fusion processing on the first image feature and the second image feature output by the backbone network, and may include at least one branch network. The spatial feature fusion network is used to perform fusion processing on the target fusion features output by the temporal feature fusion network, and can be constructed based on a pre-trained fusion network, such as a feature fusion network in a semantic segmentation model. The feature recognition network is used to perform change recognition on the target fusion features output by the spatial feature fusion network to obtain a change recognition result, and can be constructed based on a pre-trained recognition network, such as a semantic segmentation head in a semantic segmentation model. In some embodiments, at least one of the pre-trained fusion network, the pre-trained fusion network and the pre-trained recognition network is constructed based on a semantic segmentation model, for example, it may include but is not limited to FPN, FCN, ASPP, UNet, SegNet and DeepLab, such as the pre-trained fusion network is FPN, the pre-trained recognition network is DeepLabv3, etc.

[0096] In one embodiment, the backbone network, spatial domain feature fusion network and feature recognition network can all be constructed based on the existing semantic segmentation model, such as based on the same semantic segmentation model. In this way, it is equivalent to inserting a newly added time domain feature fusion network into an overall semantic segmentation model. The backbone network, spatial domain feature fusion network and feature recognition network in the model can all be pre-trained using a pre-training set for the semantic segmentation network. In this way, for the overall target recognition model, only a small amount of change detection training data is needed to fine-tune the model, so as to realize the difference information extraction learning of the time domain feature fusion network and the adjustment of model parameters, and the training cost is low.

[0097] The following is an introduction to the image change recognition method of this application in combination with the above target recognition model. Please refer to Figure 7 , the method comprises the following steps.

[0098] S301: Acquire a group of images to be identified, where the group of images to be identified includes a reference image and a corresponding comparison image.

[0099] S303: extracting features of the reference image and the comparison image respectively through the backbone network to obtain a first image feature of the reference image and a second image feature of the comparison image.

[0100] In the embodiment of the present application, the backbone network of the target recognition model is a twin network, that is, the same backbone neural network is used to construct different branches of the twin network, and the backbone neural networks share weights to extract features from the reference image and the comparison image respectively, thus ensuring high comparability between the features of the first image and the features of the second image. Please refer to Figure 8 , Figure 8 The network framework of the target recognition model in one embodiment is shown in FIG. 0 and contrast image t 1 , the backbone network extracts their features respectively, and then f 0 and f 1 Output to the time domain feature fusion network. Figure 8 In the embodiment, the backbone network is constructed based on VGG.

[0101] Specifically, the same backbone network is used to extract image features of different scales, and then generate a feature pyramid. In one embodiment, four scales of feature maps are used, which are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input scale, that is, the resolutions are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image resolution, respectively.

[0102] S305: Extract difference information from the first image feature and the second image feature through a time domain feature fusion network to obtain at least one difference feature.

[0103] In the present application embodiment, Fig. 9 As shown in the figure, there are three types of image changes, namely "appear", "disappear" and "exchange". "Appear" means that there is no object in the reference image, but it appears in the comparison image. "Disappear" means that there is an object in the reference image, but it disappears in the comparison image. "Exchange" means that the object in the reference image disappears, and a new object appears in the comparison image. Correspondingly, the difference feature categories include new features, disappearance features, exchange features and overall transformation features. Specifically, the MTF network is used to analyze the two features f 0 and f 1 Difference information is extracted to obtain at least one of a newly added feature, a disappeared feature, a swap feature and an overall transformed feature, wherein the swap feature represents a feature that exists in the reference image but disappears in the comparison image and is replaced by another object.

[0104] Please refer to Fig.10In this embodiment, a plurality of branch networks are arranged in the MTF network, including a first characteristic branch network (T 0 object info branch), the second feature branch network (T 1 The first feature branch network is used to retain the first image feature, and the second feature branch network is used to retain the second image feature, so as to provide semantic information of the original reference image and the comparison image. Different difference feature categories correspond to different difference feature branch networks, such as Fig.10 As shown, the newly added features correspond to the newly added branch network (Appearbranch), the disappeared features correspond to the disappeared branch network (Disappear branch), and the overall changed features correspond to the overall changed branch network (Exchange branch).

[0105] Furthermore, if Figure 8 In the embodiment of the invention, the newly added branch network utilizes the feature f 1 Subtract feature f 0 , and perform ReLU operation on the subtraction result, the corresponding expression is f appear =ReLU(f 1 -f 0 ), and then the new features are obtained through convolution operation (conv). The vanishing branch network uses the feature f 0 Subtract feature f 1 , and perform ReLU operation on the subtraction result, the corresponding expression is f disappear =ReLU(f 0 -f 1 ), and then the new features are obtained through convolution operation. The overall transformation branch network transforms the feature f 1 and feature f 0 Perform feature difference operation, the expression is f different =max(f 0 ,f 1 )-min(f 0 ,f 1 ), and then the overall transformation features are obtained by convolution operation. Specifically, the above convolution operation can be specifically to perform feature convolution using a 3x3 convolution layer. In addition, the newly added branch network and the disappeared branch network can share weights, and the first feature branch network and the second feature branch network can share weights.

[0106] As mentioned above, the feature scales are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input scale, that is, the input features in each branch include features of four feature scales, then f 1 -f 0 , f 1-f 0 and max(f 0 ,f 1 )-min(f 0 ,f 1 ) is to subtract the features of each feature scale to obtain the newly added features, disappeared features and overall transformation features corresponding to the scales of 1 / 4, 1 / 8, 1 / 16 and 1 / 32.

[0107] In another embodiment, the overall transformation branch network can be replaced by a switching branch network, which can transform the feature f 1 and feature f 0 Perform exchange feature extraction to obtain a fourth distinguishing feature, perform feature mapping processing (such as ReLU operation) on the fourth distinguishing feature to obtain a fourth intermediate feature, and then perform a convolution operation on the fourth intermediate feature to obtain an exchange feature.

[0108] In another embodiment, the MTF network may include six branches, namely the newly added branch network, the disappeared branch network, the exchange branch network, the overall transformation branch network, the first characteristic branch network and the second characteristic branch network.

[0109] S307: Perform feature fusion processing on the first image feature, the second image feature and at least one difference feature through a time domain feature fusion network to obtain a target fusion feature f usion .

[0110] Specifically, the MTF network performs the fusion. As shown in the figure, the feature fusion process in this embodiment is to obtain the f 1 , f 0 , f appear , f disappear and f different Add them separately, taking 1 / 4 as an example, add the 1 / 4 scale f 1 , f 0 , f appear , f disappear and f different By adding them together, we get the target fusion features at 1 / 4 scale, and finally get a feature pyramid formed by feature maps of each scale.

[0111] S309: Performing scale fusion processing on the target fusion features corresponding to each feature scale through the spatial domain feature fusion network to obtain target recognition features.

[0112] Specifically, the MSF network is used to perform scale fusion of target fusion features. When the target fusion features only include one feature scale, if the scale of the target fusion features is smaller than the input scale, the target fusion features are upsampled and convolved to reduce the dimension to obtain target recognition features. The resolution and dimension of the target recognition features are the same as the input of the target recognition model. When multi-scale target fusion features are obtained, the multi-scale target fusion features need to be scale-fused to obtain a preset number of target fusion features of the same feature scale, and then the preset number of target fusion features are convolved and reduced to obtain target recognition features. Specifically, if Fig.11 As shown, in this embodiment, the MSF network can be built based on the FPN framework. The network is stable and reliable and can be easily deployed in industrial applications. The feature pyramid input to the MSF network includes feature scales of 1 / 4, 1 / 8, 1 / 16 and 1 / 32. First, each target fusion feature in the feature pyramid is subjected to the action of a 1x1 convolution layer, and the channel is changed to 256 dimensions. The target fusion features of the smaller scale are subjected to a 2x bilinear interpolation and then added to the adjacent target fusion features of the larger scale. The obtained new feature map is further processed by a 3x3 convolution layer, and then all the convolved features are bilinearly upsampled to 1 / 4 of the original input scale, and all the 1 / 4 scale features are spliced ​​together, and then a 1x1 convolution operation is performed on the spliced ​​features to change the channel to 512 dimensions to obtain the target recognition feature.

[0113] S311: Using a feature recognition network, based on a preset semantic segmentation algorithm, changes in the target recognition features are identified to obtain a change recognition result.

[0114] Specifically, the target recognition features are input into the feature recognition network for change recognition to predict the area where the image changes occur. The feature recognition network can be constructed using a pre-trained semantic segmentation network. Figure 8 As shown, the feature recognition network in this embodiment is FCN, which uses 3x3 convolution to reduce the number of channels of the target recognition feature by 4 times, and then uses 1x1 convolution to reduce the number of channels to 2, and finally uses 4 times upsampling and softmax operation to predict the changed area to obtain the change recognition result. In another embodiment, DeepLabv3's ASPP is used for semantic segmentation processing to more effectively capture multi-scale information.

[0115] Based on all or part of the above implementation methods, an embodiment of the present application also provides a method for training a target recognition model, which specifically includes the following steps.

[0116] S401: Obtain an initial recognition model and a training sample set.

[0117] Specifically, the training sample set includes a sample image group and a corresponding sample label, and the sample label represents the image change of the sample image group. In some embodiments, the sample label is a binary image representing the real mark, that is, during the training process of the initial recognition model and the application process of the target recognition model, the model needs to perform change recognition and prediction for each pixel.

[0118] S403: Using the sample image group as the input of the initial recognition model and the sample label as the expected output, the initial recognition model is iteratively trained for image change recognition until the training iteration conditions are met.

[0119] S405: Taking the initial recognition model that meets the training iteration conditions as the target recognition model.

[0120] Specifically, the initial recognition model includes an initial backbone network, an initial time domain feature fusion network, an initial space domain feature fusion network and an initial feature recognition network. The initial backbone network, the initial space domain feature fusion network and the initial feature recognition network can be constructed based on the pre-trained network, respectively. Similar to the above, the initial time domain feature fusion network includes multiple branch networks, including at least a first image branch network, a second image branch network and a difference branch network. The difference feature classification and the number of branches corresponding to the difference branch network can be set based on actual needs. The difference branch network includes a feature mapping layer, a convolution layer and a fusion layer. Some or all of the difference branch networks also include a feature mapping layer, such as a ReLU layer. By iteratively training the initial recognition model and fine-tuning the model parameters of the initial recognition model, the model parameters are fixed after the iteration conditions are met to obtain the target recognition model.

[0121] like Figure 8 In the embodiment, the real label of the change detection data set is a binary image, and the weighted cross entropy loss is used as the loss function. The loss at each pixel is defined as follows, where p c and c are the change recognition results and sample labels of the model output respectively. w is the balance weight, n b and n c are the sizes of the unchanged and changed areas in the image, respectively.

[0122] L=-wy c logp c -(1-w)(1-y c )log(1-p c )

[0123]

[0124] For the target recognition model provided in the embodiment shown in the figure above, the change recognition result test was performed using a benchmark dataset. Please refer to Table 1 and Table 2 below. The results show that compared with other change detection methods, the change recognition method (C-3PO) of the present application achieved the best performance on the benchmark dataset. Tables 1 and 2 show the F1-score of each algorithm on the same change recognition task. The larger the score, the better the model performance and effect. Among them, GSV is the Google Street View dataset, and Average is the average F1-score for the GSV and TSUNAMI datasets. The VL-CMU-CD dataset is a portion of the images in the Visual Localization CMU (VL-CMU) dataset.

[0125] Table 1

[0126]

[0127] Table 2

[0128]

[0129] Furthermore, the MTF network can learn image changes such as "appearance", "disappearance", "exchange" and "overall change". Fig.12 , visualize the different branches in the MTF module of the target recognition model in the above embodiment, where t 0 and t 1 They are the reference image and the comparison image respectively. The “disappear” graph shows the learned disappearance change, the “appear” graph shows the learned disappearance change, and the “exchange” graph shows the learned overall difference change. “predict” is the change recognition result obtained by the target recognition model by integrating these three changes. “GT” is the sample label, which is composed of Fig.12 It can be seen that the model of the present application has a high change recognition accuracy.

[0130] In another test, please refer to Table 3 below, which summarizes the performance of using different branches in the MTF network. I, A, D, and E correspond to the four difference feature branches mentioned above. The I, A, D, and E columns in the table indicate that only one of the corresponding branches is used. I+A means that I and A branches are used at the same time. I+A+D+E means that all branches are used. For different tasks, branches can be reasonably selected. For example, in the VL-CMU-CD task, only "disappearing" changes are included, so D branch or I+D branch is used.

[0131] Table 3

[0132]

[0133] In another embodiment, referring to Table 4, MobileNet and Swin-T are used as backbone networks, FCN and DeepLabv3 are used as change recognition networks (Head), and the number of fused layers of the MSF network is adjusted. The number of fused layers represents the preset number of the aforementioned feature scales. For different tasks, increasing the number of MSF network layers, i.e., multi-scale feature extraction and difference information extraction, can improve the model effect and improve the recognition accuracy.

[0134] Table 4

[0135]

[0136] In summary, among the existing methods for solving the change detection problem using deep neural networks, EFNet modifies the U-Net model and applies it to the change detection task, CosimNet adopts the DeepLabv2 model and proposes a contrast loss function with a threshold to alleviate the noise problem, HPCFNet uses an adaptive feature fusion method to fuse feature pairs, and DR-TANet introduces the attention mechanism into the field of change detection. However, the existing methods all require a change detection network to be redesigned, and then a large amount of change detection data is used for training. The structures of these network models are becoming more and more complex and difficult to deploy in industrial applications. In addition, due to the small amount of training data for change detection, these networks are difficult to be well trained, resulting in low performance. Based on the technical solution provided by the above-mentioned application, the time domain feature fusion network, the spatial domain feature fusion network and the semantic segmentation model are applied to change recognition and detection, and can be trained by ordinary image data and a small amount of change detection data, so as to obtain higher model performance and recognition effect, and can adjust the branches in the time domain feature fusion network based on different tasks, and match different pre-trained networks, with good universality, simple model structure, stable, reliable and easy to deploy in industrial applications. In addition, the model can be built with an existing semantic segmentation model, and the model parameters can be fine-tuned based on a small amount of change detection data to obtain a target recognition model with good model transfer performance and low training cost.

[0137] The image change recognition method provided in the embodiment of the present application can be applied to a variety of change detection scenarios, such as defect detection scenarios in industrial quality inspection, map update scenarios, disaster assessment scenarios, problem recognition scenarios in the agricultural field, and object recognition scenarios in the security field. Fig.13 and Fig.14 , Fig.14 A set of images of non-defective products and defective products are shown in FIG. Based on the above change detection method, the location of the defect (the box position in the figure) can be identified. Fig.13 A set of street views of the same location at different times are shown. The changed area can be determined through the above change detection, and then the relevant geographic information can be updated.

[0138] The present application embodiment also provides an image change recognition device 800, such as Fig.15 As shown, Fig.15 A schematic diagram of the structure of an image change recognition device provided in an embodiment of the present application is shown, and the device may include the following modules.

[0139] An image group acquisition module 10 is used to acquire an image group to be identified, wherein the image group to be identified includes a reference image and a corresponding comparison image;

[0140] Feature extraction module 20: used to extract features from the reference image and the comparison image respectively, to obtain a first image feature of the reference image and a second image feature of the comparison image;

[0141] The difference feature extraction module 30 is used to extract difference information from the first image feature and the second image feature to obtain at least one difference feature;

[0142] The time domain feature fusion network 40 is used to perform feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain a target fusion feature;

[0143] The change recognition module 50 is used to recognize changes in the target fusion features based on a preset semantic segmentation algorithm, and obtain a change recognition result between the reference image and the comparison image.

[0144] In some embodiments, at least one difference feature includes a disappearance feature, and the difference feature extraction module 30 may include the following units.

[0145] A first feature subtraction unit: used for subtracting the second image feature from the first image feature to obtain a first distinguishing feature;

[0146] A first feature mapping unit: configured to perform feature mapping processing on the first distinguishing feature based on a preset mapping method to obtain a first intermediate feature;

[0147] The first convolution unit is used to perform convolution processing on the first intermediate feature to obtain the disappearing feature.

[0148] In some embodiments, at least one difference feature includes a newly added feature, and the difference feature extraction module 30 may include the following units.

[0149] A second feature subtraction unit: used for subtracting the first image feature from the second image feature to obtain a second distinguishing feature;

[0150] A second feature mapping unit: configured to perform feature mapping processing on the second distinguishing feature based on a preset mapping method to obtain a second intermediate feature;

[0151] The second convolution unit is used to perform convolution processing on the second intermediate feature to obtain a new feature.

[0152] In some embodiments, at least one difference feature includes an overall transformation feature, and the difference feature extraction module 30 may include the following units.

[0153] A feature difference processing unit: used for performing feature difference processing on the first image feature and the second image feature to obtain a third intermediate feature;

[0154] The third convolution unit is used to perform convolution processing on the third intermediate feature to obtain the overall transformation feature.

[0155] In some embodiments, the feature extraction module 20 can be specifically used to: use the same backbone network to perform multi-scale feature extraction on the reference image and the comparison image respectively to obtain a first image feature and a second image feature, and the first image feature and the second image feature respectively include image features of a preset number of feature scales.

[0156] The difference feature extraction module 30 may be specifically used to: for each feature scale, extract difference information from the first image feature and the second image feature to obtain at least one difference feature corresponding to each feature scale;

[0157] The time domain feature fusion network 40 can be specifically used to: for each feature scale, perform feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain target fusion features corresponding to each feature scale.

[0158] In some embodiments, the change identification module may specifically include the following units.

[0159] Scale fusion unit: used to perform scale fusion processing on the target fusion features corresponding to each feature scale to obtain target recognition features;

[0160] Change recognition unit: used to identify changes in target recognition features based on a preset semantic segmentation algorithm to obtain change recognition results.

[0161] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0162] An embodiment of the present application provides an image change recognition device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method provided in the above method embodiment.

[0163] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0164] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Fig.16 is a hardware structure block diagram of an electronic device for an image change recognition method provided in an embodiment of the present application. Fig.16 As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or, one or more operating systems 921, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0165] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0166] It can be understood by those skilled in the art that Fig.16 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.16 More or fewer components as shown, or with Fig.16 Different configurations are shown.

[0167] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to an image change recognition method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method provided by the above method embodiment.

[0168] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0169] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0170] It can be seen from the embodiments of the image change recognition method, device, equipment, server, terminal, storage medium and program product provided by the present application that the technical solution of the present application first obtains a group of images to be recognized, the group of images to be recognized includes a reference image and a corresponding comparison image, and extracts features from the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image; then extracts difference information from the first image feature and the second image feature to obtain at least one difference feature, and performs feature fusion processing on the first image feature, the second image feature and at least one difference feature to obtain a target fusion feature; and then performs change recognition on the target fusion feature based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image. In this way, the difference information extraction between the reference image and the comparison image is decoupled from feature fusion and feature recognition, and then network modules can be designed separately for difference information extraction, feature fusion and feature recognition, thereby improving the universality of the image change recognition method and realizing the migration of the semantic segmentation method in the change recognition task. The existing semantic segmentation network or a pre-trained semantic segmentation network with a small amount of modification can be used to realize change recognition, reduce the complexity of model design and structure, improve deployability and reduce training costs.

[0171] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for identifying image changes, characterized in that: The method comprises: Acquire a group of images to be identified, wherein the group of images to be identified includes a reference image and a corresponding comparison image; Based on the backbone network, feature extraction is performed on the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image; based on the time domain feature fusion network, difference information is extracted from the first image feature and the second image feature to obtain two or more difference features; the two or more difference features include two or more of disappearing features, newly added features, exchange features and overall transformation features; Based on the time domain feature fusion network, feature fusion processing is performed on the first image feature, the second image feature and the two or more difference features to obtain a target fusion feature; through the spatial domain feature fusion network and the feature recognition network, scale fusion and change recognition based on a preset semantic segmentation algorithm are performed on the target fusion feature to obtain a change recognition result between the reference image and the comparison image.

2. The method according to claim 1, characterized in that The two or more difference features include a disappearance feature, and the two or more difference features obtained by extracting difference information from the first image feature and the second image feature include: subtracting the second image feature from the first image feature to obtain a first distinguishing feature; Performing feature mapping processing on the first distinguishing feature based on a preset mapping method to obtain a first intermediate feature; The first intermediate feature is subjected to convolution processing to obtain the disappearing feature.

3. The method according to claim 1, characterized in that The two or more difference features include a newly added feature, and the two or more difference features obtained by extracting difference information from the first image feature and the second image feature include: subtracting the first image feature from the second image feature to obtain a second distinguishing feature; Performing feature mapping processing on the second distinguishing feature based on a preset mapping method to obtain a second intermediate feature; Perform convolution processing on the second intermediate feature to obtain the newly added feature.

4. The method according to claim 1, characterized in that: The two or more difference features include an overall transformation feature, and the two or more difference features obtained by extracting difference information from the first image feature and the second image feature include: performing feature difference processing on the first image feature and the second image feature to obtain a third intermediate feature; The third intermediate feature is subjected to convolution processing to obtain the overall transformation feature.

5. The method according to any one of claims 1 to 4, characterized in that The extracting features of the reference image and the comparison image respectively to obtain a first image feature of the reference image and a second image feature of the comparison image comprises: The reference image and the comparison image are respectively subjected to multi-scale feature extraction using the same backbone network to obtain the first image feature and the second image feature, wherein the first image feature and the second image feature respectively include image features of a preset number of feature scales.

6. The method according to claim 5, characterized in that The extracting difference information from the first image feature and the second image feature to obtain two or more difference features includes: For each feature scale, extracting difference information between the first image feature and the second image feature to obtain the two or more difference features corresponding to each feature scale; The performing feature fusion processing on the first image feature, the second image feature and the two or more difference features to obtain a target fusion feature includes: For each feature scale, feature fusion processing is performed on the first image feature, the second image feature and the two or more difference features to obtain target fusion features corresponding to each feature scale.

7. The method according to claim 6, characterized in that The performing change recognition on the target fusion feature based on a preset semantic segmentation algorithm to obtain a change recognition result between the reference image and the comparison image includes: Performing scale fusion processing on the target fusion features corresponding to each of the characteristic scales to obtain target recognition features; Based on the preset semantic segmentation algorithm, changes in the target recognition features are identified to obtain the change recognition result.

8. An image change recognition device, characterized in that: The device comprises: An image group acquisition module is used to acquire an image group to be identified, wherein the image group to be identified includes a reference image and a corresponding comparison image; Feature extraction module: used to extract features from the reference image and the comparison image based on the backbone network, respectively, to obtain a first image feature of the reference image and a second image feature of the comparison image; A difference feature extraction module is used to extract difference information from the first image feature and the second image feature based on a time domain feature fusion network to obtain two or more difference features; the two or more difference features include two or more of disappearing features, newly added features, exchange features and overall transformation features; A time domain feature fusion network: used for performing feature fusion processing on the first image feature, the second image feature and the two or more difference features based on the time domain feature fusion network to obtain a target fusion feature; Change recognition module: used to perform scale fusion and change recognition based on a preset semantic segmentation algorithm on the target fusion features through a spatial domain feature fusion network and a feature recognition network, so as to obtain a change recognition result between the reference image and the comparison image.

9. The device according to claim 8, characterized in that The two or more difference features include a disappearance feature, and the difference feature extraction module includes: A first feature subtraction unit is used to subtract the second image feature from the first image feature to obtain a first distinguishing feature; A first feature mapping unit: configured to perform feature mapping processing on the first distinguishing feature based on a preset mapping method to obtain a first intermediate feature; The first convolution unit is used to perform convolution processing on the first intermediate feature to obtain the disappearing feature.

10. The device according to claim 8, characterized in that The two or more difference features include newly added features, and the difference feature extraction module includes: A second feature subtraction unit is used to subtract the first image feature from the second image feature to obtain a second distinguishing feature; A second feature mapping unit: configured to perform feature mapping processing on the second distinguishing feature based on a preset mapping method to obtain a second intermediate feature; The second convolution unit is used to perform convolution processing on the second intermediate feature to obtain the newly added feature.

11. The device according to claim 8, characterized in that The two or more difference features include an overall transformation feature, and the difference feature extraction module includes: A feature difference processing unit: used for performing feature difference processing on the first image feature and the second image feature to obtain a third intermediate feature; The third convolution unit is used to perform convolution processing on the third intermediate feature to obtain the overall transformation feature.

12. The device according to any one of claims 8 to 11, characterized in that The feature extraction module is specifically used for: The reference image and the comparison image are respectively subjected to multi-scale feature extraction using the same backbone network to obtain the first image feature and the second image feature, wherein the first image feature and the second image feature respectively include image features of a preset number of feature scales.

13. The device according to claim 12, characterized in that The difference feature extraction module is specifically used to: for each feature scale, extract difference information from the first image feature and the second image feature to obtain the two or more difference features corresponding to each feature scale; The time domain feature fusion network is specifically used to: for each feature scale, perform feature fusion processing on the first image feature, the second image feature and the two or more difference features to obtain target fusion features corresponding to each feature scale.

14. The device according to claim 13, characterized in that The change identification module includes: A scale fusion unit: used for performing scale fusion processing on the target fusion features corresponding to each characteristic scale to obtain a target recognition feature; A change identification unit is used to identify changes in the target identification features based on the preset semantic segmentation algorithm to obtain the change identification result.

15. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method according to any one of claims 1 to 7.

16. A computer device, comprising a processor and a memory, characterized in that: The memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the image change recognition method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the image change recognition method according to any one of claims 1 to 7 is implemented.

Images