Image processing method and device and storage medium

Abstract

The invention discloses an image processing method and device and a storage medium, relates to the technical field of image recognition, and can improve the recognition accuracy of a difference region in an image difference comparison process. The method comprises the following steps: firstly, acquiring a comparison image pair comprising a first image and a second image; further, a first image feature of the first image and a second image feature of the second image are extracted, and the first image feature and the second image feature are subjected to feature fusion to obtain a fusion feature. And then decoding the fusion feature to obtain a first candidate difference region between the first image and the second image, determining an implicit feature of the fusion feature, and performing autoregression processing on the implicit feature to obtain a second candidate difference region between the first image and the second image. And determining a difference region between the first image and the second image based on the first candidate difference region and the second candidate difference region.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image recognition technology, and in particular, to an image processing method, device and storage medium. Background Art

[0002] Image difference comparison refers to the use of computer vision technology to automatically analyze and process images to detect changes, target objects or abnormal conditions in images. It is widely used in smart manufacturing, intelligent operation and maintenance, industrial monitoring, artificial intelligence (AI) inspection and other fields.

[0003] In existing image processing methods, a common practice is to perform a differential calculation between the current image to be detected and the comparison image to obtain a differential image, then segment the differential image to generate foreign body candidate areas, and finally generate the final foreign body detection result through the nearest neighborhood fusion algorithm.

[0004] However, due to factors such as shooting time and lighting conditions, the pixel difference between the current image to be detected and the comparison image may be relatively large, and the above solution is prone to false detection and missed detection. For example, when the image to be detected taken during night inspection is compared with the comparison image taken during the day, even if the content of the two images has not changed, a large number of different areas will still be detected due to the large pixel difference. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, an apparatus, and a storage medium for improving the recognition accuracy of difference areas during image difference comparison.

[0006] To achieve the above objectives, the present application provides the following technical solutions:

[0007] In a first aspect, an image processing method is provided, the method comprising: first obtaining a comparison image pair including a first image and a second image. Further, extracting a first image feature of the first image and a second image feature of the second image, and fusing the first image feature with the second image feature to obtain a fused feature. Then, decoding the fused feature to obtain a first candidate difference region between the first image and the second image, and determining an implicit feature of the fused feature, and performing autoregressive processing on the implicit feature to obtain a second candidate difference region between the first image and the second image. Based on the first candidate difference region and the second candidate difference region, determining the difference region between the first image and the second image.

[0008] The image processing method provided in the embodiment of the present application starts with the overall features of the image, compares and finds the difference area, so that the image comparison is more comprehensive, and the determined difference area is also more accurate. Specifically, the present application extracts the first image feature of the first image and the second image feature of the second image through the encoder in the trained difference area detection model. In order to facilitate the contrast restoration of the image, the present application fuses the first image feature with the second image feature to obtain the fusion feature, and uses the decoder to decode to obtain the first candidate difference area between the first image and the second image, and uses the implicit feature of the fusion feature to perform autoregression processing to obtain the second candidate difference area between the first image and the second image. Since the difference area detection model is obtained based on a plurality of sample image pairs collected in advance and trained by a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference area mark, the trained difference area detection model can comprehensively and accurately extract the complete whole of the two images, and make the feature difference of the difference part larger, while the feature of the non-difference part tends to be the same. Then, the results obtained by combining the two different restoration methods are combined to determine the difference area between the first image and the second image. Therefore, compared with the pixel comparison method used in related technologies which is greatly affected by lighting and shooting time, the image comparison method of the present application is insensitive to environments such as lighting and shadows, and can accurately detect differences in the same content and lost items.

[0009] In a possible implementation, obtaining a comparison image pair includes: obtaining an image to be detected and a comparison image corresponding to the image to be detected; using the comparison image as a reference, aligning the image to be detected and the comparison image to obtain a configured image to be detected; using the configured image to be detected as a first image and the comparison image as a second image to obtain a comparison image pair.

[0010] It can be understood that the electronic device aligns the image to be detected with the comparison image, which can geometrically align the two images acquired by different devices or at different times, making them consistent in spatial coordinates, facilitating the comparison between the image to be detected and the comparison image, and improving the accuracy of the comparison results.

[0011] In one possible implementation, extracting a first image feature of a first image and a second image feature of a second image includes: extracting the first image feature of the first image and the second image feature of the second image through an encoder of a difference area detection model; the difference area detection model is obtained based on a plurality of pre-collected sample image pairs and trained with a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference area mark.

[0012] It can be understood that the use of deep learning algorithms to train the difference area detection model can autonomously learn and extract features in the image, thereby improving the efficiency and accuracy of feature extraction.

[0013] In one possible implementation, the difference area detection model includes a first encoder and a second encoder, the first encoder and the second encoder have the same network structure and share network parameters; through the encoder in the trained difference area detection model, the first image feature of the first image and the second image feature of the second image are extracted, including: inputting the first image into the first encoder to obtain the first image feature; inputting the second image into the second encoder to obtain the second image feature.

[0014] It can be understood that the first encoder and the second encoder have the same network structure and share network parameters, which enables them to learn similar feature representations and facilitate subsequent feature fusion.

[0015] In one possible implementation, the first encoder and the second encoder both include multiple convolution blocks, and different convolution blocks are used to extract different sub-features; the first image is input into the first encoder to obtain a first image feature, including: passing the first image through the multiple convolution blocks in the first encoder in sequence to obtain multiple first sub-features, and using the multiple first sub-features as the first image feature; the second image is input into the second encoder to obtain a second image feature, including: passing the second image through the multiple convolution blocks in the second encoder in sequence to obtain multiple second sub-features, and using the multiple second sub-features as the second image feature.

[0016] It can be understood that the electronic device can extract different first sub-features and second sub-features through different convolution blocks in the first encoding and different convolution blocks in the second encoding in sequence, thereby enriching the information of the first image features and the second image features.

[0017] In a possible implementation, the first image is sequentially passed through multiple convolution blocks in the first encoder to obtain multiple first sub-features, including: for a first convolution block at any position in the first encoder, down-sampling the first image through the first convolution block to obtain an initial first sub-feature; obtaining an initial second sub-feature corresponding to the initial first sub-feature, and determining a feature difference between the initial first sub-feature and the initial second sub-feature; the initial second sub-feature is obtained by down-sampling the second image by a second convolution block at the same position as the first convolution block in the second encoder; and adding the feature difference to the initial first sub-feature to obtain a first sub-feature.

[0018] In a possible implementation, the second image is sequentially passed through multiple convolution blocks in the second encoder to obtain multiple second sub-features, including: for a second convolution block at any position in the second encoder, downsampling the second image through the second convolution block to obtain an initial second sub-feature; obtaining an initial first sub-feature corresponding to the initial second sub-feature, and determining a feature difference between the initial first sub-feature and the initial second sub-feature; the initial first sub-feature is obtained by downsampling the first image by a first convolution block at the same position as the second convolution block in the first encoder; and adding the feature difference to the initial second sub-feature to obtain a second sub-feature.

[0019] It can be understood that for the initial first sub-features and the initial second sub-features obtained by sampling two convolution blocks at the same position, adding the feature differences between the two can make the first sub-features and the second sub-features learned by the difference area detection model more accurate, such as making the difference between two sub-features with large feature differences tend to be larger, and the difference between two sub-features with small feature differences tend to be the same.

[0020] In a possible implementation, the first image feature and the second image feature are feature-fused, including: obtaining a fusion weight of each first sub-feature, and fusing each first sub-feature with the corresponding second sub-feature according to the fusion weight.

[0021] It is understandable that in order to better extract the difference between the apparent information and semantic information of the image, the fusion features of the image at multiple levels and scales can be obtained through feature fusion. By setting the fusion weight, the fusion can be performed in a hierarchical manner at different scales, so that the impact of single factor changes (such as pixel changes caused by factors such as illumination changes) on the image comparison results is less robust. Among them, the scale is the size of the feature map, and the level corresponds to the convolution layer in the encoder. The features of different convolution layers express different semantic information, and one convolution layer represents a level.

[0022] In one possible implementation, autoregression processing is performed on implicit features to obtain a second candidate difference region between the first image and the second image, including: passing the implicit features through an implicit autoregression module in a difference region detection model to obtain a reconstructed image and a likelihood probability of each pixel value in the reconstructed image; and determining the region in the reconstructed image corresponding to the pixel value whose likelihood probability is greater than or equal to a preset probability as the second candidate difference region.

[0023] It is understandable that when an electronic device uses an implicit autoregressive module to model data, there is no need to sort the features. Compared with the traditional autoregressive model, the reconstruction condition is more flexible because it does not depend on the order of the features.

[0024] In a possible implementation, the method further includes: marking the difference areas on the first image and the second image respectively to obtain a first marked image and a second marked image; inputting the first marked image and the second marked image into a trained discriminant model to obtain a discriminant result; the discriminant result is used to reflect the reliability of the difference area; based on the discriminant result, determining the target difference area from the difference area; the reliability of the target difference area is greater than or equal to a preset degree.

[0025] It can be understood that the electronic device uses the discriminant model to perform secondary judgment on the difference area detected by the difference area detection model to generate the final difference area detection result, which can effectively suppress the false alarm phenomenon of the difference area detection.

[0026] In a second aspect, an image processing device is provided, which includes an acquisition unit, a processing unit and a determination unit; the acquisition unit is used to acquire a comparison image pair; the comparison image pair includes a first image and a second image; the processing unit is used to extract a first image feature of the first image and a second image feature of the second image; the processing unit is also used to fuse the first image feature with the second image feature to obtain a fused feature; the processing unit is also used to decode the fused feature to obtain a first candidate difference area between the first image and the second image; the processing unit is also used to determine an implicit feature of the fused feature and perform autoregression processing on the implicit feature to obtain a second candidate difference area between the first image and the second image; the determination unit is used to determine the difference area between the first image and the second image based on the first candidate difference area and the second candidate difference area.

[0027] In a third aspect, the present application provides an electronic device, which includes a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0028] In a fourth aspect, the present application provides a computer program product. When the computer program product is executed in an electronic device, the electronic device executes the related method described in the first aspect to implement the method described in the first aspect.

[0029] In a fifth aspect, the present application provides a readable storage medium, which includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0030] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect and will not be elaborated on again. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the composition of an image processing system provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0033] Figure 3 One of the flowcharts of the image processing method provided in the embodiment of the present application;

[0034] Figure 4 A schematic diagram of implicit feature extraction provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a difference area detection model provided in an embodiment of the present application;

[0037] Figure 7 The second flowchart of the image processing method provided in the embodiment of the present application;

[0038] Figure 8 A schematic diagram of secondary verification provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of the difference area detection effect provided in the embodiment of the present application;

[0040] Fig.10 A schematic diagram of the inspection process of the railway image provided in the embodiment of the present application;

[0041] Fig.11 A schematic diagram of the composition of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] In the description of this application, unless otherwise specified, " / " indicates that the objects associated with each other are in an "or" relationship, for example, A / B can represent A or B; "and / or" in this application is merely a description of the association relationship between associated objects, indicating that three relationships may exist, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0043] In the description of this application, unless otherwise specified, "plurality" means two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0045] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0046] It is understood that the "embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It is understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0047] It can be understood that in the present application, "when" and "if" both mean that corresponding processing will be carried out under certain objective circumstances, and do not limit the time, nor do they require any judgment action when implementing, nor do they mean the existence of other limitations.

[0048] It can be understood that some optional features in the embodiments of the present application may be implemented independently in certain scenarios without relying on other features, such as the solution on which they are currently based, to solve corresponding technical problems and achieve corresponding effects, or may be combined with other features according to needs in certain scenarios. Accordingly, the devices provided in the embodiments of the present application may also realize these features or functions accordingly, which will not be elaborated here.

[0049] In this application, unless otherwise specified, the same or similar parts between the various embodiments can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The implementation methods of this application described below do not constitute a limitation on the scope of protection of this application.

[0050] In order to make the embodiments of the present application clearer, some concepts involved in the present application are first briefly introduced.

[0051] 1. Computer vision technology.

[0052] Computer vision is a science that studies how to make machines "see". To put it more specifically, computer vision can be called machine vision, which means using cameras and computers to replace human eyes to identify, track and measure targets to obtain images, and further perform graphic processing so that the computer processes the images into images that are more suitable for human eye observation or transmission to instruments for detection.

[0053] 2. Machine learning.

[0054] Machine learning is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0055] 3. Image difference comparison.

[0056] Image difference comparison refers to the use of computer vision technology to automatically analyze and process images to detect changes, target objects or abnormal conditions in images. Image difference comparison is widely used in smart manufacturing, smart operation and maintenance, industrial monitoring, artificial intelligence (AI) inspection and other fields.

[0057] 4. Image inspection.

[0058] Image inspection refers to comparing the image to be inspected with the standard image to achieve the purpose of inspecting the image to be inspected.

[0059] Traditional image inspection methods usually require operation and maintenance personnel to regularly visit each facility deployment point to check and compare with standard facility images to determine whether the equipment is abnormal or whether items at the deployment point are missing. This inspection method has high labor costs, low inspection efficiency, and is prone to omissions. By introducing computer vision and machine learning algorithms, intelligent image inspection can be achieved, that is, intelligent devices can automatically identify abnormal conditions, target objects or changes in images to provide timely warnings and decision support for operation and maintenance personnel. In contrast, the use of intelligent image inspection methods can effectively reduce labor costs, greatly improve the efficiency and accuracy of inspections, reduce errors caused by human factors, and provide more comprehensive inspection coverage. In addition, intelligent image inspection can also be combined with other technologies, such as the Internet of Things, big data analysis, etc., to achieve more intelligent inspection and monitoring. By combining image data with other data sources for analysis, more comprehensive problem identification and fault prediction can be achieved, providing more reference for operation and maintenance decisions. To sum up, intelligent image inspection has broad application prospects and practical value. By automatically analyzing and processing image data, it can improve inspection efficiency, reduce costs, and bring greater convenience and development opportunities to the fields of intelligent manufacturing and industrial monitoring.

[0060] In some related technologies, the method of intelligent image inspection is to perform differential calculation on the image to be detected and the comparison image to obtain a differential image; then the differential image is segmented to generate foreign body candidate areas; finally, the final foreign body detection result is generated through the nearest neighborhood fusion algorithm.

[0061] However, optionally, due to factors such as shooting time and lighting conditions, the pixel difference between the image to be detected and the comparison image may be relatively large. For example, the image to be detected is an infrared image taken during a night inspection, and the comparison image is a visible light image taken during the day under the same scene. When comparing the image differences, even if the content of the two images has not changed, a large number of difference areas will still be detected due to the large pixel difference between the two images, causing the scheme to fail. In addition, when there are shadows and other phenomena that affect the image quality in the image to be detected, the scheme will also have serious false detections and missed detections.

[0062] In other related technologies, the method of intelligent image inspection is to first find the similar object area in the two images, and determine the area as the candidate feature area, then perform geometric transformation and denoising on the candidate feature area to generate aligned candidate feature areas, and finally perform pixel comparison and difference area contour recognition on the candidate feature areas in the two images to obtain the difference area. Since this scheme requires the same content in the two images to determine the candidate feature area, it is difficult for this scheme to detect the difference area when there is a missing object or shadow, resulting in false detection and missed detection.

[0063] In summary, the existing technical solutions are easily prone to false detection and missed detection due to the factors such as the great influence of lighting, shooting time and limited inspection scenes.

[0064] In response to the above problems, an embodiment of the present application provides an image processing method which, during the image inspection process, makes full use of various aspects of the image features and cascades and fuses these features, so that the pixel changes caused by factors such as illumination changes have a less robust effect on the image comparison results, thereby improving the recognition accuracy of the difference areas.

[0065] The image processing method provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. It should be noted that the embodiments of the present application can refer to each other, for example, the same or similar steps, method embodiments, system embodiments and device embodiments can refer to each other without limitation.

[0066] The image processing method provided in the embodiment of the present application can be applied to an image processing system. Figure 1 FIG. 4 shows a structural diagram of the image processing system. Figure 1 As shown, the image processing system 10 includes an image acquisition device 11 and an image processing device 12. The image acquisition device 11 and the image processing device 12 may be connected via a wired network or a wireless network.

[0067] The image acquisition device 11 can be used to acquire a scene image of a target scene; wherein the scene image can be a visible light image or other types of images (such as a grayscale image, a thermal imaging image, etc.).

[0068] The image acquisition device 11 may be a camera or an internet protocol camera (IPC) etc. arranged at a specific location.

[0069] The image acquisition device 11 may also be a dual-light device. In this case, the lens of the image acquisition device 11 includes a visible light lens and an infrared thermal imaging lens, which are respectively used to acquire visible light and infrared light of the target scene.

[0070] In some embodiments, the image acquisition device 11 may also be used to send the acquired image to the image processing device 12 .

[0071] As described above, the image acquisition device 11 and the image processing device 12 may be connected via a wired network or a wireless network. The wired network or the wireless network may include one or more media or devices capable of transmitting image data from the image acquisition device 11 to the image processing device 12 .

[0072] In some embodiments, the wired or wireless network may include one or more communication media that enable the image acquisition device 11 to send image data directly to the image processing device 12 in real time. In this embodiment, the image acquisition device 11 may modulate the image data according to a communication standard (e.g., a wireless communication protocol) and send the modulated image data to the image processing device 12. The one or more communication media may include wireless, and / or, wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. Optionally, the one or more communication media may form part of a packet-based network, which may be, for example, a local area network, a wide area network, or a global network (e.g., the Internet). Optionally, the one or more communication media may also include routers, switches, base stations, or other devices that facilitate communication from the image acquisition device 11 to the image processing device 12.

[0073] The image processing device 12 can be used to perform difference comparison on the images collected by the image collection device 11 to determine the difference area. The specific difference comparison process can refer to the image processing method introduced in the following method embodiment, which will not be repeated here.

[0074] The image processing device 12 may be an electronic device having an image processing function, such as a computer or a server.

[0075] Among them, the server can be a single server, or it can be a server cluster consisting of multiple servers. In some implementations, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, for example, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-cloud, and a multi-cloud, etc., or any combination thereof. The embodiments of the present application are not limited to this.

[0076] It should be noted that the above Figure 1 In the description, the image acquisition device 11 and the image processing device 12 are described as independent devices. Optionally, the image acquisition device 11 and the image processing device 12 can also be combined into one device. For example, the image acquisition device 11 or its corresponding function, and the image processing device 12 or its corresponding function can be integrated into one device. This embodiment of the application is not limited to this.

[0077] The execution subject of the image processing method provided in the embodiment of the present application may be the above-mentioned image processing device 12. As mentioned above, the image processing device 12 may be an electronic device with computing and processing functions, such as a computer or a server. Optionally, the image processing device 12 may also be a processor (such as a central processing unit (CPU)) in the aforementioned electronic device; or, the image processing device 12 may also be an application (application, APP) installed in the aforementioned electronic device with a passenger-carrying behavior detection function; or, the image processing device 12 may also be a functional module in the aforementioned electronic device with a passenger-carrying behavior detection function, etc. The embodiment of the present application does not limit this.

[0078] For simplicity of description, the following description will be made by taking the image processing device 12 as an electronic device as an example.

[0079] Figure 2 The following is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application. Figure 2 As shown, the electronic device may include: a processor 20 , a memory 21 , a communication line 22 , a communication interface 23 , and an input / output interface 24 .

[0080] The processor 20 , the memory 21 , the communication interface 23 and the input / output interface 24 may be connected via a communication line 22 .

[0081] The processor 20 is used to execute the instructions stored in the memory 21 to implement the image processing method provided in the following embodiments of the present application. The processor 20 can be a CPU, a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / single-chip microcomputer, a programmable logic device (PLD) or any combination thereof. The processor 20 can also be any other device with processing functions, such as a circuit, a device or a software module, which is not limited in the embodiments of the present application. In one example, the processor 20 may include one or more CPUs, such as Figure 2 As an optional implementation, the electronic device may include multiple processors, for example, in addition to the processor 20, it may also include a processor 25 ( Figure 2 The dashed line is used as an example.

[0082] The memory 21 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 21 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage devices, etc., and the embodiments of the present application are not limited to this.

[0083] It should be noted that the memory 21 may exist independently of the processor 20, or may be integrated with the processor 20. The memory 21 may be located inside the electronic device, or may be located outside the electronic device, which is not limited in the embodiment of the present application.

[0084] The communication line 22 is used to transmit information between the components included in the electronic device.

[0085] The communication interface 23 is used to communicate with other devices (such as the above-mentioned image acquisition device 11) or other communication networks. The other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 23 can be a module, a circuit, a transceiver or any device capable of achieving communication.

[0086] The input / output interface 24 is used to implement human-computer interaction between a user and an electronic device, for example, to implement action interaction or information interaction between a user and an electronic device.

[0087] Exemplarily, the input / output interface 24 may be a mouse, a keyboard, a display screen, or a touch display screen, etc. Action interaction or information interaction between a user and the electronic device may be achieved through a mouse, a keyboard, a display screen, or a touch display screen, etc.

[0088] It should be noted that Figure 2 The structure shown in the figure does not constitute a limitation on the electronic device, except Figure 2 In addition to the components shown, the electronic device may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0089] Combine the following Figure 1 The system shown introduces the image processing method provided in the embodiment of the present application.

[0090] Figure 3 The flowchart of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 2 The electronic device of the hardware structure shown is executed, such as Figure 3 As shown, the method includes S301 to S306.

[0091] S301: Obtain a comparison image pair.

[0092] The comparison image pair includes a first image and a second image. The first image and the second image may be any two images, and the embodiment of the present application does not limit the first image and the second image. For example, the first image may be an image to be detected, and the second image may be a comparison image corresponding to the image to be detected.

[0093] As a possible implementation manner, the electronic device may acquire the image to be detected through an image acquisition device, and use the pre-stored image as a comparison image to obtain a comparison image pair.

[0094] It should be noted that the comparison image can be pre-stored in the electronic device by the operation and maintenance personnel, or can be collected by the electronic device through an image collection device and pre-stored in the device itself.

[0095] Optionally, in order to obtain a comparison image, the electronic device can use an image captured by an image acquisition device deployed at the target location at a historical time point as a comparison image. Further, the electronic device uses an image captured by an image acquisition device at the same location at the current time point as the image to be detected.

[0096] As another possible implementation, after the electronic device obtains the image to be detected and the comparison image corresponding to the image to be detected, the image to be detected and the comparison image are registered with each other based on the comparison image to obtain the configured image to be detected. Furthermore, the electronic device can use the configured image to be detected as the first image and the comparison image as the second image to obtain a comparison image pair.

[0097] Image registration refers to the process of aligning two or more images so that they completely or approximately overlap in space. Common image registration methods include feature point-based registration, region-based registration, phase-correlation-based registration, etc. The embodiments of the present application do not limit the specific configuration method.

[0098] Feature point-based registration refers to the process of extracting feature points from an image, such as corners and edges, and then matching these feature points to achieve image registration. Region-based registration refers to the process of matching some regions in an image to achieve image registration. Phase correlation-based registration refers to the process of calculating the Fourier transform of two images and then calculating their cross-correlation function to achieve image registration.

[0099] Optionally, the image involved in the present application includes multiple pixel points or multiple feature points. Based on the comparison image, the registration processing of the image to be detected and the comparison image may include: the electronic device extracts feature points of the image to be detected and the comparison image respectively, and matches the extracted feature points to obtain the positional relationship between each key point, for example, the transformation matrix of the image to be detected can be calculated through the matching results. Furthermore, the electronic device rotates the image to be detected to be consistent with the comparison image through the transformation matrix, thereby achieving geometric alignment.

[0100] It can be understood that the electronic device aligns the image to be detected with the comparison image so that the two images acquired by different devices or at different times are geometrically aligned and consistent in spatial coordinates, thereby facilitating the comparison between the image to be detected and the comparison image and improving the accuracy of the comparison results.

[0101] S302: Extract first image features of the first image and second image features of the second image.

[0102] As a possible implementation manner, the electronic device may input the first image and the second image into an encoder in a pre-trained difference area detection model respectively to obtain a first image feature of the first image and a second image feature of the second image.

[0103] The difference region detection model is obtained based on a plurality of sample image pairs collected in advance and trained by a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference region mark. Optionally, the difference region detection model includes an encoder, a decoder, and an autoregressor. The following is an introduction to the various parts of the difference region detection module:

[0104] The encoder can be composed of a multi-layer neural network, which is used to map the input data to a low-dimensional representation. This low-dimensional representation can be regarded as an abstraction of the original data, and usually has better interpretability and stronger expression ability. In the image difference area detection task, the encoder can convert the input image into a vector or matrix, which contains the main feature information of the input image, such as features of visual information such as color, texture, shape (also known as appearance information) and features of semantic information reflecting the content and structure of the image. In other words, the input of the encoder is an image, and the output of the encoder is the image features of the image.

[0105] In the present application, the image features of an image may include, but are not limited to, visual features such as the color of the image, the texture of the image, the shape of the image (also referred to as appearance information), and may also include semantic features that reflect the semantic information of the content and structure of the image. For example, the first image features extracted by the embodiment of the present application may include not only the visual features of the first image but also the semantic features of the first image, and the second image features may include the visual features of the second image and the semantic features of the second image.

[0106] The decoder is the opposite of the encoder and is used to map the low-dimensional representation back to the high-dimensional space to generate the output. Optionally, the decoder uses the same architecture as the encoder and uses the back-propagation algorithm to update the weights and biases. In the image difference area detection task, the decoder starts from the encoded low-dimensional vector or matrix to generate a new image that can reflect the difference area between the first image and the second image.

[0107] Among them, the autoregressor is used for image autoregression, which uses an autoregressive model to generate realistic images. The autoregressive model is essentially a generative model that predicts the next pixel value by using the conditional probability distribution of the current pixel value. In image autoregression, the autoregressive model traverses each pixel of the feature image starting from the upper left corner and uses the generated pixels to predict the probability distribution of the next pixel.

[0108] Optionally, the electronic device can initially construct a difference area detection module, and train the constructed difference area detection model based on multiple sample image pairs collected in advance to obtain a trained difference area detection model. Specifically, for each sample image pair, it includes a first sample image and a second sample image with a difference area mark. When the electronic device trains the difference area detection model, it uses the first sample image and the second sample image as input, and uses the difference area marked between the first sample image and the second sample image as a label, and continuously adjusts the parameters of the encoder, decoder and autoregressor according to the output result of the difference area detection model until the number of training reaches a preset number or the output result converges. Then, the electronic device can accurately extract the first image features of the first image and the second image features of the second image through the encoder in the trained difference area detection model.

[0109] S303: Fusing the first image feature with the second image feature to obtain a fused feature.

[0110] As a possible implementation, the electronic device may perform feature fusion on the first image feature and the second image feature through a feature fusion device in the difference region detection model to obtain a fusion feature. Specifically, when performing feature fusion on the first image feature and the second image feature, the feature fusion device may directly superimpose the first image feature and the second image feature to obtain a superimposed image feature, and use the superimposed image feature as the fusion feature.

[0111] As another possible implementation method, when the electronic device fuses the first image feature with the second image feature through the feature fusion device in the difference area detection model, it can also directly weighted average the first image feature and the second image feature, and use the weighted average image feature as the fusion feature.

[0112] It should be noted that the first image feature and the second image feature may include not only the visual features of the image but also the semantic features of the image. Therefore, the first image feature may include multiple first sub-features, and the second image feature may include multiple second sub-features. The number of first sub-features is the same as the number of second sub-features, and they correspond one to one. For example, the first image feature includes three first sub-features, which are respectively used to reflect the color information, texture information, and content information of the first image. Then, the second image feature also includes three second sub-features, which are respectively used to reflect the color information, texture information, and content information of the second image.

[0113] Optionally, when performing feature fusion on the first image feature and the second image feature, the electronic device may obtain the fusion weight of each first sub-feature, and perform feature fusion on each first sub-feature and the corresponding second sub-feature according to the fusion weight. Alternatively, the electronic device may obtain the fusion weight of each second sub-feature, and perform feature fusion on each first sub-feature and the corresponding second sub-feature according to the fusion weight.

[0114] Exemplarily, among the three first sub-features, the fusion weight of the first sub-feature (represented by x1) used to reflect the color information of the first image is 0.1, the fusion weight of the first sub-feature (represented by x2) used to reflect the texture information of the first image is 0.5, and the fusion weight of the first sub-feature (represented by x3) used to reflect the texture information of the first image is 0.8. Feature fusion is performed with the three second sub-features (represented by y1, y2, y3) to obtain the fusion feature: 0.1(x1+y1)+0.5(x2+y2)+0.8(x3+y3).

[0115] In some embodiments, the operation and maintenance personnel can also specify the fusion strength (such as strong, medium, weak) between each sub-feature in the feature fusion stage, so that the electronic device can fuse the first sub-feature with the second sub-feature according to the corresponding fusion strength. For example, if the fusion strength of x1 and y1 is set to weak, the electronic device may not fuse x1 and y1, that is, x1 and y2 may be discarded, so that the fused feature may not contain x1 and y1. If the fusion strength of x3 and y3 is set to strong, the electronic device can focus on fusing x3 and y3. If the fusion strength of x1 and y1, x2 and y2, and x3 and y3 are all strong, the fusion features obtained by the electronic device will include all the information of x1-x3 and y1-y3, and high-intensity fusion will also make the recognition accuracy of the difference area detection model higher.

[0116] It is understandable that in order to better extract the difference between the apparent information and semantic information of the image, the fusion features of the image at multiple levels and scales can be obtained through feature fusion. By setting the fusion weight, the fusion can be performed in a hierarchical manner at different scales, so that the impact of single factor changes (such as pixel changes caused by factors such as illumination changes) on the image comparison results is less robust. Among them, the scale is the size of the feature map, and the level corresponds to the convolution layer in the encoder. The features of different convolution layers express different semantic information, and one convolution layer represents a level.

[0117] S304: Decode the fused features to obtain a first candidate difference region between the first image and the second image.

[0118] As a possible implementation method, the electronic device inputs the fused features after fusing the first image features with the second image features into a decoder in a trained difference area detection model. The decoder can output the difference area between the two input images, and the electronic device can use the difference area as the first candidate difference area between the first image and the second image.

[0119] It should be noted that the difference area output by the decoder may include an area where the position, shape, etc. of an object in the two images are changed. In practical applications, the difference area decoded by the decoder may be an image of the same size as the first image or the second image, and the grayscale value of the difference area in the image is different from the grayscale value of other non-difference areas.

[0120] S305 : Determine implicit features of the fused features, and perform autoregression processing on the implicit features to obtain a second candidate difference region between the first image and the second image.

[0121] As a possible implementation, the electronic device may perform feature conversion on the fused feature to obtain an implicit feature of the fused feature. Further, the electronic device performs autoregressive processing on the implicit feature to obtain a second candidate difference region between the first image and the second image.

[0122] It should be noted that the embodiments of the present application do not limit the specific method of obtaining implicit features. For example, the electronic device may convolve the fused features to obtain implicit features, and may also use the Cartesian product between the first image feature and the second image feature as the implicit feature. Implicit features are features obtained after processing by convolution or full connection, etc., and are mainly used to reflect the positional relationship between pixels in the feature image. Optionally, the electronic device may pass the implicit features through an implicit autoregression module in a trained difference region detection model to obtain a reconstructed image and the likelihood probability of each pixel value in the reconstructed image. Furthermore, the electronic device determines the area corresponding to the pixel value in the reconstructed image whose likelihood probability is greater than or equal to the preset probability as the second candidate difference area.

[0123] It is understandable that when an electronic device uses an implicit autoregressive module to model data, there is no need to sort the features. Compared with the traditional autoregressive model, the reconstruction condition is more flexible because it does not depend on the order of the features.

[0124] like Figure 4As shown, the fused features are converted into implicit features and then sent to the implicit autoregressive module to fully explore the local difference information between the contrasting images and enhance the comparison of the differences in the entire image. During the training process of the difference area detection model, the electronic device reconstructs the image based on the implicit features. Specifically, when there is no difference between the two images, the implicit autoregressive module can correctly reconstruct the image and output the likelihood probability of each pixel as zero. When there is a difference in the image, the implicit autoregressive module cannot correctly reconstruct the image, and the likelihood probability of the pixels in the difference area is 1. Therefore, in the application process, the electronic device can directly regress the implicit features to obtain the likelihood probability of each pixel value, and determine the difference area according to the likelihood probability.

[0125] S306: Determine a difference region between the first image and the second image based on the first candidate difference region and the second candidate difference region.

[0126] As a possible implementation manner, the electronic device may determine both the first candidate difference region and the second candidate difference region as the difference region between the first image and the second image.

[0127] As another possible implementation, the electronic device determines the difference area that exists in both the first candidate difference area and the second candidate difference area as the difference area between the first image and the second image. In other words, the intersection area or the overlapping area of ​​the first candidate difference area and the second candidate difference area is determined as the difference area between the first image and the second image.

[0128] As another possible implementation manner, the electronic device may screen out difference regions that meet the screening conditions from the first candidate difference regions and the second candidate difference regions based on a preset screening condition, and determine the difference regions as the difference regions between the first image and the second image.

[0129] Optionally, after generating the first candidate difference region, the decoder of the difference region detection model will also output the confidence of each first candidate difference region, and after generating the second candidate difference region, the implicit autoregressive module of the difference region detection model will also output the confidence of each second candidate difference region. Therefore, the screening condition can be set to a confidence greater than or equal to a preset threshold. The size of the confidence can reflect the credibility of the difference region between the two images.

[0130] Exemplarily, the confidence of the first candidate difference region 1 is 0.5, the confidence of the first candidate difference region 2 is 0.8, the confidence of the second candidate difference region 1 is 0.3, and the confidence of the second candidate difference region 2 is 0.9, the preset threshold can be set to 0.7, and the electronic device can determine the first candidate difference region 2 and the second candidate difference region 2 as the difference region between the first image and the second image. The first candidate difference region 1 and the second candidate difference region 1 can be the same or different, and the first candidate difference region 2 and the second candidate difference region 2 can be the same or different.

[0131] The image processing method provided in the embodiment of the present application starts with the overall features of the image, compares and finds the difference area, so that the image comparison is more comprehensive, and the determined difference area is also more accurate. Specifically, the present application extracts the first image feature of the first image and the second image feature of the second image through the encoder in the trained difference area detection model. In order to facilitate the contrast restoration of the image, the present application fuses the first image feature with the second image feature to obtain the fusion feature, and uses the decoder to decode to obtain the first candidate difference area between the first image and the second image, and uses the implicit feature of the fusion feature to perform autoregression processing to obtain the second candidate difference area between the first image and the second image. Since the difference area detection model is obtained based on a plurality of sample image pairs collected in advance and trained by a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference area mark, the trained difference area detection model can comprehensively and accurately extract the complete whole of the two images, and make the feature difference of the difference part larger, while the feature of the non-difference part tends to be the same. Then, the results obtained by combining the two different restoration methods are combined to determine the difference area between the first image and the second image. Therefore, compared with the pixel comparison method used in related technologies which is greatly affected by lighting and shooting time, the image comparison method of the present application is insensitive to environments such as lighting and shadows, and can accurately detect differences in the same content and lost items.

[0132] In one design, the encoder in the above-mentioned difference area detection model may include a first encoder and a second encoder, and the first encoder and the second encoder have the same network structure and share network parameters. Optionally, the electronic device may input the first image into the first encoder to obtain the first image feature, and input the second image into the second encoder to obtain the second image feature.

[0133] It can be understood that the first encoder and the second encoder have the same network structure and share network parameters, which enables them to learn similar feature representations and facilitate subsequent feature fusion.

[0134] like Figure 5As shown, the first encoder and the second encoder each include a plurality of convolution blocks, wherein different convolution blocks are used to extract different sub-features. The electronic device may sequentially pass the first image through the plurality of convolution blocks in the first encoder to obtain a plurality of first sub-features, and use the plurality of first sub-features as first image features. Similarly, the electronic device may sequentially pass the second image through the plurality of convolution blocks in the second encoder to obtain a plurality of second sub-features, and use the plurality of second sub-features as second image features.

[0135] It can be understood that the electronic device can extract different first sub-features and second sub-features through different convolution blocks in the first encoding and different convolution blocks in the second encoding in sequence, thereby enriching the information of the first image features and the second image features.

[0136] like Figure 5 As shown, for the first convolution block at any position in the first encoder, the electronic device can downsample the first image through the first convolution block to obtain an initial first sub-feature. The electronic device obtains the initial second sub-feature corresponding to the initial first sub-feature, and determines the feature difference between the initial first sub-feature and the initial second sub-feature. The initial second sub-feature is obtained by downsampling the second image by the second convolution block at the same position as the first convolution block in the second encoder. Further, the electronic device adds the feature difference to the initial first sub-feature to obtain a first sub-feature.

[0137] Alternatively, for the second convolution block at any position in the second encoder, the electronic device can downsample the second image through the second convolution block to obtain an initial second sub-feature. The electronic device obtains the initial first sub-feature corresponding to the initial second sub-feature, and determines the feature difference between the initial first sub-feature and the initial second sub-feature. The initial first sub-feature is obtained by downsampling the first image by the first convolution block at the same position as the second convolution block in the first encoder. Further, the electronic device adds the feature difference to the initial second sub-feature to obtain a second sub-feature.

[0138] It can be understood that for the initial first sub-features and the initial second sub-features obtained by sampling two convolution blocks at the same position, adding the feature differences between the two can make the first sub-features and the second sub-features learned by the difference area detection model more accurate, such as making the difference between two sub-features with large feature differences tend to be larger, and the difference between two sub-features with small feature differences tend to be the same.

[0139] In addition, if Figure 5 As shown, the decoder may include multiple convolution blocks. In contrast to the encoder, the convolution blocks in the decoder are used to upsample the fused features. The number of convolution blocks in the decoder and the encoder may be the same or different, which is not limited in the present embodiment.

[0140] like Figure 6 As shown, in the difference area detection model, the first encoder and the second encoder can adopt a twin encoding branch structure, which consists of two parts, upper and lower parts, and shares the same network parameters to effectively extract the shared features of the image. The electronic device first aligns the two images to obtain a comparison image pair. The electronic device inputs the first image in the comparison image pair into the first encoder to obtain the first image feature, and the second image into the second encoder to obtain the second image feature. Furthermore, the electronic device fuses the first image feature with the second image feature to obtain a fused feature, and passes the fused feature through the decoder branch and the autoregressive branch respectively, and outputs the difference area between the first image and the second image.

[0141] In one design, in order to further improve the accuracy of difference region detection and reduce false positives, e.g. Figure 7 As shown, after the above S305, the image processing method provided in the embodiment of the present application may further include:

[0142] S401 : Mark different regions on a first image and a second image respectively to obtain a first marked image and a second marked image.

[0143] As a possible implementation manner, the electronic device determines the difference area between the first image and the second image based on the above S305, and marks the corresponding positions on the first image and the second image respectively to obtain a first marked image and a second marked image.

[0144] S402: Input the first labeled image and the second labeled image into a discrimination model to obtain a discrimination result.

[0145] The discrimination result is used to reflect the reliability of the difference area.

[0146] Optionally, the discriminant model may be composed of an encoder and a decoder. For example, the discriminant model may be a two-way discriminator, that is, the two-way discriminator includes two encoders, and the two encoders are used to extract features of the first marked image and features of the second marked image respectively.

[0147] In some embodiments, the electronic device can train the constructed discriminant model based on multiple sample labeled image pairs to obtain a trained discriminant model. Specifically, for each sample labeled image pair, it includes a first sample labeled image and a second sample labeled image with difference area marks, and these difference area marks can be true or false. When the electronic device trains the discriminant model, it uses the first sample labeled image and the second sample labeled image as input, and uses the truth or falsehood of the difference area marked between the first sample image and the second sample image as a label, and continuously adjusts the parameters of the encoder and decoder according to the discrimination result of the discriminant model until the number of training times reaches a preset number or the output result converges. Then the electronic device can input the first labeled image and the second labeled image into the trained discriminant model to obtain a discrimination result.

[0148] like Figure 8 As shown, the electronic device inputs the first marked image and the second marked image into a dual-path discriminator to perform secondary verification on the difference areas detected by the difference area detection model, and outputs the discrimination results of each difference area.

[0149] S403: Determine a target different region from the different regions based on the discrimination result.

[0150] The reliability of the target difference area is greater than or equal to a preset reliability.

[0151] As a possible implementation, the electronic device uses the discrimination result of the difference region as the reliability, and compares each discrimination result with a preset degree. Further, the electronic device determines the difference region whose discrimination result is greater than or equal to the preset degree as the target difference region.

[0152] It can be understood that the electronic device uses the discriminant model to perform secondary judgment on the difference area detected by the difference area detection model to generate the final difference area detection result, which can effectively suppress the false alarm phenomenon of the difference area detection.

[0153] like Fig. 9 As shown, the image processing method provided in the embodiment of the present application can successfully detect the difference between any two images, wherein the difference area has been marked in each image. According to statistics, the difference area detected by the image processing method provided in the embodiment of the present application can reach 100% accuracy.

[0154] The present application embodiment does not limit the specific application scenario. For example, the image processing method provided in the present application embodiment can be applied to the scene of railway image inspection. Fig.10As shown, the camera located at the iron scaffolding position point can monitor the objects at the preset points around the iron scaffolding to determine whether there are changes and losses. Specifically, the operation and maintenance personnel can set inspection points for the camera to instruct the camera to take inspection images at the inspection points, and store the captured inspection images in the image library, wherein a pair of inspection images includes a first inspection image taken by the camera at a first time, and a second inspection image taken at a second time, that is, the first inspection image and the second inspection image correspond to the same inspection point position, but the shooting time is different. The electronic device can obtain a pair of inspection images through the image library interface of the camera, and perform image comparison on the pair of inspection images, including image registration, difference area extraction, and discrimination model discrimination, etc. The specific image comparison process can refer to the above embodiment, which will not be repeated here. Furthermore, the electronic device can mark the detected difference area in the inspection image, and send the marked inspection image to the camera at the inspection point to instruct the camera to store the marked inspection image in the image library. In addition, if the electronic equipment detection detects that there are different areas in the pair of inspection images after image comparison, the camera at the inspection point can also be instructed to generate an alarm to alert the operation and maintenance personnel.

[0155] The image processing method provided by the embodiment of the present application can also be applied in the industrial production process to detect product defects, surface damage or assembly errors, etc. The electronic device can quickly detect the difference between the images by comparing the real-time captured image with the comparison image, and trigger the corresponding alarm or automatic control. For example, the electronic device uses the comparison product image and the real-time captured product image as a comparison image pair (wherein the real-time captured product image can be the first image, and the comparison product image can be the second image), and extracts the first image feature of the first image and the second image feature of the second image through the encoder in the difference area detection model. Further, the electronic device fuses the first image feature with the second image feature to obtain a fused feature, and decodes the fused feature through the decoder in the difference area detection model to obtain a first candidate difference area between the first image and the second image, and determines the implicit feature of the fused feature, and performs autoregressive processing on the implicit feature to obtain a second candidate difference area between the first image and the second image. The electronic device determines the difference area between the first image and the second image based on the first candidate difference area and the second candidate difference area.

[0156] The image processing method provided in the embodiment of the present application can also be applied in security scenarios to detect abnormal events, such as intrusion, theft or vandalism. By comparing the real-time image captured by the camera with the reference image, the electronic device can quickly discover changes that are inconsistent with the expected scene and take corresponding security measures in a timely manner. For example, the electronic device can use the comparison monitoring image and the real-time monitoring image as a comparison image pair (wherein the real-time monitoring image can be the first image, and the comparison monitoring image can be the second image), and extract the first image feature of the first image and the second image feature of the second image through the encoder in the difference area detection model. Further, the electronic device fuses the first image feature and the second image feature to obtain a fused feature, and decodes the fused feature through the decoder in the difference area detection model to obtain a first candidate difference area between the first image and the second image, and determine the implicit feature of the fused feature, and perform autoregressive processing on the implicit feature to obtain a second candidate difference area between the first image and the second image. The electronic device determines the difference area between the first image and the second image based on the first candidate difference area and the second candidate difference area.

[0157] The image processing method provided in the embodiment of the present application can also be applied in building scenes to monitor the health status and deformation of building structures. By comparing the regularly taken images with the reference images, the electronic device can detect changes or damages in the building structure and provide real-time monitoring and reporting. For example, the electronic device can use the comparison building image and the real-time taken building image as a comparison image pair (wherein the real-time taken building image can be the first image and the comparison building image can be the second image), and extract the first image feature of the first image and the second image feature of the second image through the encoder in the difference area detection model. Further, the electronic device fuses the first image feature with the second image feature to obtain a fused feature, and decodes the fused feature through the decoder in the difference area detection model to obtain a first candidate difference area between the first image and the second image, and determines the implicit feature of the fused feature, and performs autoregressive processing on the implicit feature to obtain a second candidate difference area between the first image and the second image. The electronic device determines the difference area between the first image and the second image based on the first candidate difference area and the second candidate difference area.

[0158] The image processing method provided in the embodiment of the present application can also be applied in traffic management scenarios to monitor traffic flow, vehicle violations or traffic accidents. The electronic device can quickly identify abnormal situations and trigger corresponding traffic control or alarms by comparing the image captured by the traffic camera with the reference image. For example, the electronic device can use the contrast traffic image and the real-time captured traffic image as a contrast image pair (wherein the real-time captured traffic image can be the first image, and the contrast traffic image can be the second image), and extract the first image feature of the first image and the second image feature of the second image through the encoder in the difference area detection model. Further, the electronic device fuses the first image feature with the second image feature to obtain a fused feature, and decodes the fused feature through the decoder in the difference area detection model to obtain a first candidate difference area between the first image and the second image, and determine the implicit feature of the fused feature, and perform autoregressive processing on the implicit feature to obtain a second candidate difference area between the first image and the second image. The electronic device determines the difference area between the first image and the second image based on the first candidate difference area and the second candidate difference area.

[0159] The above mainly introduces the scheme of the embodiment of the present application from the perspective of method. It is understandable that, in order to realize the above functions, the management device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0160] The embodiment of the present application can divide the functional units of the management device according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0161] In an exemplary embodiment, the present application also provides an image processing device. Fig.11 Schematic diagram of the composition of the image processing device provided in the embodiment of the present application. Fig.11 As shown, the image processing device includes: an acquisition unit 501, a processing unit 502 and a determination unit 503.

[0162] The acquisition unit 501 is used to acquire a comparison image pair; the comparison image pair includes a first image and a second image; the processing unit 502 is used to extract a first image feature of the first image and a second image feature of the second image; the processing unit 502 is also used to fuse the first image feature with the second image feature to obtain a fused feature; the processing unit 502 is also used to decode the fused feature to obtain a first candidate difference region between the first image and the second image; the processing unit 502 is also used to determine an implicit feature of the fused feature and perform autoregression processing on the implicit feature to obtain a second candidate difference region between the first image and the second image; the determination unit 503 is used to determine the difference region between the first image and the second image based on the first candidate difference region and the second candidate difference region.

[0163] In a possible implementation, the acquisition unit 501 is specifically used to: acquire the image to be detected and the comparison image corresponding to the image to be detected; based on the comparison image, align the image to be detected and the comparison image to obtain a configured image to be detected; use the configured image to be detected as the first image and the comparison image as the second image to obtain a comparison image pair.

[0164] In one possible implementation, the processing unit 502 is specifically used to: extract the first image feature of the first image and the second image feature of the second image through the encoder of the difference area detection model; the difference area detection model is obtained based on a plurality of pre-collected sample image pairs and trained with a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference area mark.

[0165] In one possible implementation, the difference area detection model includes a first encoder and a second encoder, the first encoder and the second encoder have the same network structure and share network parameters; the processing unit 502 is specifically used to: input the first image into the first encoder to obtain the first image feature; input the second image into the second encoder to obtain the second image feature.

[0166] In one possible implementation, the first encoder and the second encoder both include multiple convolution blocks, and different convolution blocks are used to extract different sub-features; the processing unit 502 is specifically used to: sequentially pass the first image through the multiple convolution blocks in the first encoder to obtain multiple first sub-features, and use the multiple first sub-features as first image features; the processing unit 502 is specifically used to: sequentially pass the second image through the multiple convolution blocks in the second encoder to obtain multiple second sub-features, and use the multiple second sub-features as second image features.

[0167] In a possible implementation, the processing unit 502 is specifically used to: for a first convolution block at any position in the first encoder, downsample the first image through the first convolution block to obtain an initial first sub-feature; obtain an initial second sub-feature corresponding to the initial first sub-feature, and determine the feature difference between the initial first sub-feature and the initial second sub-feature; the initial second sub-feature is obtained by downsampling the second image through a second convolution block at the same position as the first convolution block in the second encoder; and add the feature difference to the initial first sub-feature to obtain a first sub-feature.

[0168] In a possible implementation, the processing unit 502 is specifically used to: for a second convolution block at any position in the second encoder, downsample the second image through the second convolution block to obtain an initial second sub-feature; obtain an initial first sub-feature corresponding to the initial second sub-feature, and determine the feature difference between the initial first sub-feature and the initial second sub-feature; the initial first sub-feature is obtained by downsampling the first image by the first convolution block at the same position as the second convolution block in the first encoder; and add the feature difference to the initial second sub-feature to obtain a second sub-feature.

[0169] In a possible implementation, the processing unit 502 is specifically configured to: obtain a fusion weight of each first sub-feature, and perform feature fusion on each first sub-feature and the corresponding second sub-feature according to the fusion weight.

[0170] In one possible implementation, the processing unit 502 is specifically used to: obtain a reconstructed image and a likelihood probability of each pixel value in the reconstructed image through an implicit autoregressive module in a difference region detection model; and determine the region in the reconstructed image corresponding to the pixel value whose likelihood probability is greater than or equal to a preset probability as the second candidate difference region.

[0171] In a possible implementation, the determination unit 503 is further used to: mark the difference area on the first image and the second image respectively to obtain a first marked image and a second marked image; input the first marked image and the second marked image into a trained discriminant model to obtain a discrimination result; the discrimination result is used to reflect the reliability of the difference area; based on the discrimination result, determine the target difference area from the difference area; the reliability of the target difference area is greater than or equal to a preset degree.

[0172] It should be noted that Fig.11 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or in the form of software functional units.

[0173] In an exemplary embodiment, the embodiment of the present application further provides a readable storage medium, including software instructions, which, when executed on an electronic device, enables the electronic device to execute any one of the methods provided in the above embodiments.

[0174] In an exemplary embodiment, the embodiment of the present application further provides a computer program product including computer-executable instructions, which, when executed on an electronic device, enables the electronic device to execute any one of the methods provided in the above embodiments.

[0175] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer-executable instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that contains one or more servers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid state disk (SSD)).

[0176] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other changes to the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0177] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0178] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a comparison image pair; the comparison image pair includes a first image and a second image; extracting a first image feature of the first image and a second image feature of the second image; fusing the first image feature with the second image feature to obtain a fused feature; Decoding the fused features to obtain a first candidate difference region between the first image and the second image; Determining an implicit feature of the fused feature, and performing autoregression processing on the implicit feature to obtain a second candidate difference region between the first image and the second image; Based on the first candidate different region and the second candidate different region, a different region between the first image and the second image is determined.

2. The method according to claim 1, characterized in that: The obtaining of the comparison image pair comprises: Acquire an image to be detected and a comparison image corresponding to the image to be detected; Based on the comparison image, the image to be detected is registered with the comparison image to obtain a configured image to be detected; The configured image to be detected is used as the first image, and the comparison image is used as the second image to obtain the comparison image pair.

3. The method according to claim 1 or 2, characterized in that: The extracting a first image feature of the first image and a second image feature of the second image includes: The first image features of the first image and the second image features of the second image are extracted through the encoder of the difference area detection model; the difference area detection model is obtained based on a plurality of sample image pairs collected in advance and trained with a deep learning algorithm, and each sample image pair includes a first sample image and a second sample image with a difference area mark.

4. The method according to claim 3, characterized in that The difference area detection model includes a first encoder and a second encoder, the first encoder and the second encoder have the same network structure and share network parameters; extracting a first image feature of the first image and a second image feature of the second image through the encoder in the trained difference area detection model includes: Inputting the first image into the first encoder to obtain the first image feature; The second image is input into the second encoder to obtain the second image features.

5. The method according to claim 4, characterized in that The first encoder and the second encoder each include a plurality of convolution blocks, and different convolution blocks are used to extract different sub-features; The step of inputting the first image into the first encoder to obtain the first image feature includes: Passing the first image through a plurality of convolution blocks in the first encoder in sequence to obtain a plurality of first sub-features, and using the plurality of first sub-features as the first image features; The step of inputting the second image into the second encoder to obtain the second image feature includes: The second image is sequentially passed through a plurality of convolution blocks in the second encoder to obtain a plurality of second sub-features, and the plurality of second sub-features are used as the second image features.

6. The method according to claim 5, characterized in that The step of sequentially passing the first image through a plurality of convolution blocks in the first encoder to obtain a plurality of first sub-features includes: For a first convolution block at any position in the first encoder, down-sample the first image through the first convolution block to obtain an initial first sub-feature; Acquire an initial second sub-feature corresponding to the initial first sub-feature, and determine a feature difference between the initial first sub-feature and the initial second sub-feature; the initial second sub-feature is obtained by downsampling the second image by a second convolution block in the second encoder having the same position as the first convolution block; The feature difference is added to the initial first sub-feature to obtain a first sub-feature.

7. The method according to claim 5, characterized in that The step of sequentially passing the second image through a plurality of convolution blocks in the second encoder to obtain a plurality of second sub-features includes: For a second convolution block at any position in the second encoder, down-sampling the second image through the second convolution block to obtain an initial second sub-feature; Acquire an initial first sub-feature corresponding to the initial second sub-feature, and determine a feature difference between the initial first sub-feature and the initial second sub-feature; the initial first sub-feature is obtained by downsampling the first image by a first convolution block in the first encoder having the same position as the second convolution block; The feature difference is added to the initial second sub-feature to obtain a second sub-feature.

8. The method according to claim 5, characterized in that The step of fusing the first image feature with the second image feature includes: A fusion weight of each first sub-feature is obtained, and each first sub-feature is fused with a corresponding second sub-feature according to the fusion weight.

9. The method according to claim 3, characterized in that: The performing autoregressive processing on the implicit features to obtain a second candidate difference region between the first image and the second image includes: Passing the implicit features through an implicit autoregressive module in the difference region detection model to obtain a reconstructed image and a likelihood probability of each pixel value in the reconstructed image; An area corresponding to a pixel value in the reconstructed image whose likelihood probability is greater than or equal to a preset probability is determined as the second candidate difference area.

10. The method according to claim 1 or 2, characterized in that: The method further comprises: Marking the difference area on the first image and the second image respectively to obtain a first marked image and a second marked image; Inputting the first labeled image and the second labeled image into a trained discrimination model to obtain a discrimination result; the discrimination result is used to reflect the reliability of the difference area; Based on the discrimination result, a target difference region is determined from the difference regions; and the reliability of the target difference region is greater than or equal to a preset reliability.

11. An image processing device, characterized in that: The device comprises an acquisition unit, a processing unit and a determination unit; The acquisition unit is used to acquire a comparison image pair; the comparison image pair includes a first image and a second image; The processing unit is used to extract a first image feature of the first image and a second image feature of the second image; The processing unit is further used to fuse the first image feature with the second image feature to obtain a fused feature; The processing unit is further configured to decode the fused feature to obtain a first candidate difference region between the first image and the second image; The processing unit is further used to determine an implicit feature of the fused feature, and perform autoregressive processing on the implicit feature to obtain a second candidate difference region between the first image and the second image; The determining unit is configured to determine a difference region between the first image and the second image based on the first candidate difference region and the second candidate difference region.

12. An electronic device, characterized in that: include: Processor and memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 10.

13. A readable storage medium, characterized in that: The readable storage medium includes: software instructions; When the software instructions are executed in an electronic device, the electronic device implements the method according to any one of claims 1 to 10.

Images