Method, apparatus and storage medium for detecting abnormal object

Abstract

A method, device and storage medium for detecting an abnormal object are disclosed. The method comprises: capturing an image of a scene; calculating a difference image between the captured image and a reference image, the difference image indicating pixel-level differences between the captured image and the reference image; extracting, by a first encoder, a second encoder and a third encoder of a neural network, a plurality of feature maps having a plurality of sizes for the reference image, the captured image and the difference image respectively; fusing the feature maps having the same size extracted by the encoders respectively, the fused feature maps being input to a decoder of the neural network; generating, by the decoder, a change map having the same size as the captured image based on the fused feature maps, a value of each pixel in the change map indicating whether a pixel in the captured image has a semantic change with respect to a corresponding pixel in the reference image; and identifying an abnormal object appearing in the scene based on the change map.

Description

Technical Field

[0001] This disclosure relates generally to image processing, and more specifically to methods, apparatus and storage media for detecting anomalous objects in images. Background Technology

[0002] Various technologies have been proposed to ensure public safety, one important aspect of which is monitoring specific environments to detect anomalous objects. For example, monitoring unusual objects on roads, such as animals, snowdrifts, or abandoned items. Anomalous object detection is a challenging task. On one hand, anomalous objects vary greatly in type, shape, size, and other characteristics, making them difficult to define accurately. On the other hand, collecting data on various anomalous objects and manually labeling them is extremely time-consuming and labor-intensive. Therefore, existing research on anomalous object detection often treats anomalous objects as unknowns, i.e., anomalous objects are categories not present in the training data.

[0003] Current anomaly detection methods employ deep learning-based image generation and segmentation techniques, based on the understanding that since training data does not include anomalies, image segmentation models using the training data can only segment known objects, and image generation models using the training data can only generate known objects. Then, the generated images are compared with the original images; regions not reconstructed correspond to anomalies. Such a detection method (DUIR) is described in the paper "Detecting the Unexpected via Image Resynthesis" by KrzysztofLis et al. (IEEE / CVF International Conference on Computer Vision (ICCV) Proceedings, 2019).

[0004] Other detection methods consider the model's output and are based on the understanding that when the model predicts the probability that a specific pixel in an image belongs to various known categories, the predicted classification probability is relatively high if the specific pixel corresponds to a known object, and relatively low if the specific pixel corresponds to an unknown object. Therefore, the pixel corresponding to an unknown anomalous object can be determined based on the predicted probability, which is less than a certain threshold, thus determining the location of the anomalous object in the image.

[0005] The advantage of existing detection methods is that they can use a single image to determine the location of anomalies. However, these single-image-based anomaly detection methods have poor generalization and robustness and are easily affected by the environment, resulting in missed or false identifications. Summary of the Invention

[0006] To address the aforementioned technical problems, this disclosure proposes a novel anomaly detection scheme that utilizes low-level differences between two images to identify anomalies through a neural network with a novel structure. This scheme exhibits significantly improved robustness and generalization.

[0007] According to one aspect of the present invention, a method for detecting anomalous objects using a neural network is provided, comprising: capturing an image of a scene; calculating a difference image between the captured image and a reference image, wherein the difference image indicates pixel-level differences between the captured image and the reference image; extracting multiple feature maps of multiple sizes for the reference image by a first encoder of the neural network, extracting multiple feature maps of the multiple sizes for the captured image by a second encoder of the neural network, and extracting multiple feature maps of the multiple sizes for the difference image by a third encoder of the neural network; fusing the feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder, and inputting the fused feature map into a decoder of the neural network; generating a change map of the same size as the captured image by the decoder based on the fused feature map, wherein the value of each pixel in the change map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image; and identifying anomalous objects appearing in the scene based on the change map.

[0008] According to another aspect of the present invention, an apparatus for detecting anomalous objects using a neural network is provided, comprising: a memory storing a computer program; and a processor configured to perform the following operations by executing the computer program: calculating a difference image between an image captured for a scene and a reference image, wherein the difference image indicates pixel-level differences between the captured image and the reference image; causing a first encoder of the neural network to extract a plurality of feature maps having a plurality of sizes for the reference image; causing a second encoder of the neural network to extract a plurality of feature maps having the plurality of sizes for the captured image; causing a third encoder of the neural network to extract a plurality of feature maps having the plurality of sizes for the difference image; fusing the feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder, and inputting the fused feature map into a decoder of the neural network; causing the decoder to generate a change map of the same size as the captured image based on the fused feature map, wherein the value of each pixel in the change map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image; and identifying anomalous objects appearing in the scene based on the change map.

[0009] According to another aspect of the present invention, a storage medium storing a computer program is provided, which, when executed by a computer, causes the computer to perform the method for detecting abnormal objects as described above. Attached Figure Description

[0010] Figure 1 The framework of an anomaly object detection scheme according to this disclosure is illustrated schematically.

[0011] Figure 2 It shows Figure 1 Examples of the various images in the document.

[0012] Figure 3 The architecture of a neural network according to this disclosure is illustrated schematically.

[0013] Figure 4 A flowchart illustrating a method for detecting anomalous objects according to this disclosure is shown schematically.

[0014] Figures 5A-5C A comparison of the identification results of the scheme according to this disclosure with those of the prior art DUIR is shown.

[0015] Figure 6 An exemplary configuration block diagram of computer hardware implementing the present invention is shown. Detailed Implementation

[0016] Figure 1 The framework of an anomaly object detection scheme according to this disclosure is illustrated schematically. Figure 2 It shows Figure 1 Examples of the various images in the document.

[0017] like Figure 1 As shown, a difference image is generated based on a predetermined reference image and a currently captured image. The reference image and the captured image can be images taken at different times for the same scene. Preferably, the reference image and the captured image are registered to facilitate the calculation of the difference image. If the reference image and the captured image are not registered, appropriate image processing techniques can be applied to the captured image to register it with the reference image. Given that image registration methods are well-known in the art, their specific description will be omitted in this disclosure. Furthermore, in this disclosure, the scene may include, but is not limited to, roads, waterways, buildings, courtyards, cities, farmland, etc. The reference image and the captured image may be, for example, images taken by a camera device or remote sensing images taken by a satellite.

[0018] A difference image indicates the difference between each pixel in a captured image and the corresponding pixel in a reference image. This low-feature-based difference often includes variations due to changes in light intensity, shadows, weather, etc., so misidentification is likely to occur if changes in an object are detected solely based on the difference image. However, because this low-feature-based difference also includes differences due to actual changes in the object itself, it can also be utilized.

[0019] As an example of a method for generating difference images, this disclosure employs the Structural Similarity (SSIM) algorithm. Difference images generated using this method can reflect the structural differences between two images and can partially avoid differences caused by factors such as illumination. The SSIM method is specifically described in the paper "Image quality assessment: from errorvisibility to structural similarity" by Zhou Wang et al. (IEEE Transactions on Image Processing, 2004, 13(4)). Furthermore, those skilled in the art can use other appropriate techniques to compute difference images, and this disclosure does not limit such methods.

[0020] A reference image, a captured image, and a difference image are input to an encoder 100 of a neural network. The encoder 100 extracts features for each image and fuses the extracted features. The fused features are input to a decoder 200 of the neural network, which decodes the input fused features and generates a change map. The change map reflects whether each pixel in the captured image has undergone semantic changes relative to each pixel in the reference image, and the regions in the image where semantic changes have occurred. In this disclosure, a semantic change of a pixel can be understood as a change in the category of the pixel. For example, there are pixel-level changes (e.g., changes in color or brightness) between different images of the same road on sunny and rainy days, but no semantic changes. There is a semantic change between an image of a clean road and an image of a road with fallen objects because the category of the pixel corresponding to the fallen object changes from "road" to "object".

[0021] Specifically, the transformation map can be a binary image and has the same dimensions as both the reference image and the captured image. A pixel value of "1" in the transformation map indicates that a pixel in the captured image has undergone a semantic change relative to its corresponding pixel in the reference image, while a pixel value of "0" indicates that a pixel in the captured image has not undergone a semantic change relative to its corresponding pixel in the reference image. Conversely, a pixel value of "1" indicates no semantic change, and a pixel value of "0" indicates a semantic change.

[0022] Then, based on the areas in the change map that show semantic changes, the corresponding areas in the captured image can be identified as anomalous objects.

[0023] Figure 2 Examples of reference images, captured images, difference images, and variation diagrams are shown. For example... Figure 2As shown, the reference image is a pre-captured image of the road taken by a fixed camera device, and it does not contain any objects other than the road. The captured image is a current capture of the same road taken by the same camera device, and it contains objects such as vehicles and fallen items. The difference image reflects the structural differences between the reference and captured images. The variation diagram shows the areas where semantic changes occur, which makes it easy to identify the presence and location of new objects on the road, thus identifying these objects as anomalous objects in the scene.

[0024] However, in some cases, the semantically changing regions shown in the change map may correspond to objects known to the monitor (e.g., vehicles traveling on a road), which are not of interest to the monitor. Therefore, preferably, the change regions corresponding to known objects can be filtered out in the change map to facilitate more accurate detection of unknown objects (e.g., dropped items).

[0025] Specifically, one or more detectors for detecting one or more known objects can be applied to an captured image to detect known objects in the captured image. For example, a vehicle detector can be used to detect vehicle objects in the captured image. Those skilled in the art can implement such object detectors using various existing techniques, the specific description of which is omitted here as these are well-known in the art. After a known object is detected, the pixels corresponding to the detected known object can be set to values ​​(e.g., "0") in the change map to indicate that no semantic change has occurred. In this way, the updated change map will no longer indicate change areas corresponding to normal objects (such as vehicles), but only change areas corresponding to anomalous objects (such as dropped items). This change map allows for more accurate identification of unknown anomalous objects appearing in the scene.

[0026] Figure 3 The architecture of a neural network according to this disclosure is illustrated schematically. For example... Figure 3 As shown, the neural network has an encoder-decoder structure, more specifically, it includes three encoders and decoders connected to them. A reference image is input to the first encoder 310, the captured image is input to the second encoder 320, and the difference image is input to the third encoder 330. Each of encoders 310, 320, and 330 includes a separate convolutional module and three groups, each group including a downsampling layer (max pooling layer) and a convolutional module. For clarity, Figure 3 Only the specific structure of the first group is shown; the second and third groups have the same structure. Preferably, the three encoders do not share parameters.

[0027] The decoder 400 includes convolutional layers, sigmoid activation layers, and three groups, each containing an upsampling layer and a convolutional module. Figure 3 Only the specific structure of the first group is shown; the second and third groups have the same structure.

[0028] In each encoder, a separate convolutional module extracts feature maps from the corresponding input image. The extracted feature maps are fed into a first group, where they undergo downsampling operations at the downsampling layer and then operations at the convolutional module. The feature maps output from the convolutional modules in the first group are reduced in size and fed into a second group. The second group performs the same operations as the first group, and then feeds the further reduced-size feature maps into a third group. The third group performs the same operations.

[0029] Furthermore, the feature maps output by the third group of the three encoders have the same size and are fused together. The fused feature map (hereinafter referred to as the "third fused feature map") is input to the upsampling layer in the first group of the decoder 400. The feature maps output by the second group of the three encoders have the same size and are fused together. The fused feature map (hereinafter referred to as the "second fused feature map") is input to the first group of the decoder 400 for the concatenation operation described later. The feature maps output by the first group of the three encoders have the same size and are fused together. The fused feature map (hereinafter referred to as the "first fused feature map") is input to the second group of the decoder 400 for the concatenation operation. The feature maps output by the individual convolutional modules of the three encoders have the same size and are fused together. The fused feature map (hereinafter referred to as the "fourth fused feature map") is input to the third group of the decoder 400 for the concatenation operation. Preferably, feature map fusion can be achieved by concatenating the feature maps.

[0030] In the first group of decoder 400, the third fused feature map is upsampled by an upsampling layer. The size of the upsampled feature map is increased to the same size as the second fused feature map from the encoder. This upsampled feature map is concatenated with the second fused feature map, and the concatenated feature map is input to the convolutional module in the first group. Feature decoding is performed by the convolutional module, and the output of the convolutional module is input to the upsampling layer in the second group.

[0031] In the second group of decoder 400, the feature map output from the first group is upsampled by an upsampling layer. The size of the upsampled feature map is increased to the same size as the first fused feature map from the encoder. This upsampled feature map is concatenated with the first fused feature map, and the concatenated feature map is input to the convolutional module in the second group. Feature decoding is performed by the convolutional module, and the output of the convolutional module is input to the upsampling layer in the third group.

[0032] In the third group of decoder 400, the feature map output from the second group is upsampled by an upsampling layer. The size of the upsampled feature map is increased to the same size as the fourth fused feature map from the encoder. This upsampled feature map is concatenated with the fourth fused feature map, and the concatenated feature map is input to the convolutional module in the third group. Feature decoding is performed by the convolutional module, and the output of the convolutional module is input to a 1×1 convolutional layer in decoder 400.

[0033] After processing the 1×1 convolutional layer and sigmoid activation layer, the decoder 400 outputs a variation map with the same size as the captured image.

[0034] As an example, the convolutional module of each of the encoders 310-330 and the decoder 400 can consist of a 3×3 convolutional layer, batch normalization, ReLU activation layer, or a 3×3 convolutional layer, batch normalization, and ReLU activation layer. Furthermore, for example, the kernel size of the downsampling layer can be 2×2, and the kernel size of the upsampling layer can also be 2×2.

[0035] It should be noted that the neural networks based on this disclosure are not limited to... Figure 3 The structure shown can be modified technically by those skilled in the art. As a more general limitation, in this disclosure, each encoder may include N sets of downsampling layers and convolutional modules, and correspondingly, the decoder may include N sets of upsampling layers and convolutional modules, where N is a natural number greater than 2.

[0036] The following describes a training method for a neural network based on publicly available information. The neural network can be trained using a training image set based on a cross-entropy loss function. The training image set may include mutually registered captured image training samples and reference image training samples, as well as labeled variation maps corresponding to each pair of reference image samples and captured image samples. The cross-entropy loss function can be expressed by the following mathematical formula (1):

[0037]

[0038] in, Let p represent the true label for the j-th pixel in the i-th pair of reference image samples and captured image samples, indicating whether the j-th pixel of the captured image sample has undergone a semantic change relative to the j-th pixel of the reference image sample. j denoted by , where represents the probability of a semantic change predicted by the neural network for the j-th pixel, n represents the number of reference image samples and captured image sample pairs, H represents the image height, and W represents the image width.

[0039] The following combination Figure 4This is a flowchart describing a method for detecting abnormal objects according to this disclosure. For example... Figure 4 As shown, in step S410, a difference image is calculated between the reference image and the captured image. The reference image and the captured image can be images taken at different times for the same scene, and the difference image indicates the pixel-level difference between the captured image and the reference image. Preferably, the reference image is a pre-captured image of the scene that does not contain any objects, and the reference image is aligned with the captured image.

[0040] In step S420, the first encoder of the neural network extracts multiple feature maps of multiple sizes for the reference image, the second encoder extracts multiple feature maps of multiple sizes for the captured image, and the third encoder extracts multiple feature maps of multiple sizes for the difference image.

[0041] In step S430, the feature maps with the same size extracted by the first encoder, the second encoder and the third encoder are fused together, and the fused feature map is input into the decoder of the neural network.

[0042] In step S440, the decoder generates a change map of the same size as the captured image based on the input fused feature map. Preferably, this change map is a binary map, where the value of each pixel indicates whether a pixel in the captured image has undergone a semantic change relative to the corresponding pixel in the reference image.

[0043] In step S450, abnormal objects appearing in the scene are identified based on the change map. Preferably and optionally, a detector for known objects can be used to detect known objects in the captured image, and the change map can be modified according to the detection results. Then, abnormal objects are identified based on the modified change map.

[0044] Table 1 shows the performance evaluation of the scheme and comparative examples according to this disclosure. Based on the evaluation metrics F1 score (DICE) and IoU scores, it can be seen that the scheme of this disclosure achieves more accurate recognition results.

[0045] [Table 1]

[0046]

[0047]

[0048] Figures 5A-5C The figures show the recognition results of the solution according to this disclosure and the prior art DUIR, which uses only a single image (captured image), under different scene or lighting conditions. It can be seen from the figures that the object recognition capability of the solution according to this disclosure is significantly superior to DUIR.

[0049] The methods described in the above embodiments can be implemented by software, hardware, or a combination of software and hardware. Programs included in the software can be pre-stored in a storage medium located internally or externally to the device. As an example, during execution, these programs are written to random access memory (RAM) and executed by a processor (e.g., a CPU) to implement the various methods and processes described herein.

[0050] Figure 6 An example configuration block diagram of computer hardware is shown, which can be used to implement the neural network according to the present disclosure and the apparatus for detecting abnormal objects using the neural network.

[0051] like Figure 6 As shown, in computer 600, central processing unit (CPU) 601, read-only memory (ROM) 602 and random access memory (RAM) 603 are connected to each other via bus 604.

[0052] The input / output interface 605 is further connected to the bus 604. The input / output interface 605 is connected to the following components: an input unit 606 formed by a keyboard, mouse, microphone, etc.; an output unit 607 formed by a display, speaker, etc.; a storage unit 608 formed by a hard disk, non-volatile memory, etc.; a communication unit 609 formed by a network interface card (such as a local area network (LAN) card, modem, etc.); and a driver 610 for driving a removable medium 611, such as a disk, optical disk, magneto-optical disk, or semiconductor memory.

[0053] In a computer with the above structure, the CPU 601 loads the program stored in the storage unit 608 into the RAM 603 via the input / output interface 605 and the bus 604, and executes the program to perform the method described above.

[0054] The program to be executed by the computer (CPU 601) can be recorded on a portable medium 611, which is formed as a packaging medium, such as a magnetic disk (including a floppy disk), an optical disk (including a compact optical disk-read-only memory (CD-ROM)), a digital multifunction optical disk (DVD), etc.), a magneto-optical disk, or a semiconductor memory. Furthermore, the program to be executed by the computer (CPU 601) can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0055] When the removable medium 611 is installed in the driver 610, the program can be installed in the storage unit 608 via the input / output interface 605. Alternatively, the program can be received by the communication unit 609 via a wired or wireless transmission medium and installed in the storage unit 608. Alternatively, the program can be pre-installed in the ROM 602 or the storage unit 608.

[0056] A program executed by a computer may be a program that performs processing in the order described in this specification, or it may be a program that performs processing in parallel or when needed (such as when invoked).

[0057] The units or devices described herein are for logical purposes only and do not strictly correspond to physical devices or entities. For example, the function of each unit described herein may be implemented by multiple physical entities, or the function of multiple units described herein may be implemented by a single physical entity. Furthermore, the features, components, elements, steps, etc., described in one embodiment are not limited to that embodiment, but can also be applied to other embodiments, such as replacing specific features, components, elements, steps, etc., in other embodiments, or in combination with them.

[0058] The scope of this invention is not limited to the specific embodiments described herein. Those skilled in the art will understand that various modifications or variations can be made to the embodiments described herein, depending on design requirements and other factors, without departing from the principles and spirit of the invention. The scope of this invention is defined by the appended claims and their equivalents.

[0059] Postscript:

[0060] (1) A method for detecting abnormal objects using a neural network, comprising:

[0061] To capture an image of a scene;

[0062] Calculate a difference image between a captured image and a reference image, wherein the difference image indicates pixel-level differences between the captured image and the reference image;

[0063] The first encoder of the neural network extracts multiple feature maps of multiple sizes for the reference image, the second encoder of the neural network extracts multiple feature maps of the multiple sizes for the captured image, and the third encoder of the neural network extracts multiple feature maps of the multiple sizes for the difference image.

[0064] The feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder are fused together, and the fused feature map is input into the decoder of the neural network.

[0065] The decoder generates a transformation map of the same size as the captured image based on the fused feature map, wherein the value of each pixel in the transformation map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image; and

[0066] The abnormal objects appearing in the scene are identified based on the change graph.

[0067] (2) According to the method of (1), wherein the reference image is a pre-captured image of the scene excluding abnormal objects, and the reference image is registered with the captured image.

[0068] (3) According to the method in (1), wherein the change map is a binary map, and a pixel value of 1 in the binary map indicates that a pixel in the captured image has undergone a semantic change relative to the corresponding pixel in the reference image, and a pixel value of 0 indicates that a pixel in the captured image has not undergone a semantic change relative to the corresponding pixel in the reference image.

[0069] (4) The method according to (3) further includes:

[0070] One or more detectors for detecting one or more known objects are applied to the captured image to detect known objects in the captured image;

[0071] Based on the detection results, the values ​​of pixels corresponding to the detected known objects are set to 0 in the change map; and

[0072] The updated change graph is used to identify anomalous objects in the scene.

[0073] (5) The method according to (1) further includes:

[0074] The first encoder, the second encoder, and the third encoder respectively extract N feature maps of N sizes for the corresponding image, wherein the sizes of the N feature maps decrease in order, and N is a natural number greater than 2;

[0075] The feature maps of the same size extracted by the first encoder, the second encoder and the third encoder are fused together, and the resulting N fused feature maps are input into the decoder.

[0076] The decoder performs N-1 upsampling operations based on the received N fused feature maps.

[0077] In the first upsampling, the decoder performs upsampling on the Nth fused feature map with the smallest received size;

[0078] In each of the second to N-1th upsampling operations, the decoder performs upsampling on the feature map that has been generated by splicing the following feature maps: the previous feature map obtained from the previous upsampling operation, and a fused feature map of the same size as the previous feature map.

[0079] (6) The method according to (1) further includes:

[0080] The neural network is trained using a training image set based on a cross-entropy loss function, wherein the training image set includes mutually registered training images and reference training images, and corresponding labeled change maps; and

[0081] The abnormal object is detected using a trained neural network.

[0082] (7) According to the method described in (1), wherein the scenario is a location or area that needs to be monitored.

[0083] (8) The method according to (1), wherein the scenario includes at least one of road, waterway, building, courtyard, city, and farmland.

[0084] (9) The method according to (1), wherein the reference image and the captured image are images of the scene captured by a fixed camera device at different times, or

[0085] The reference image and the captured image are remote sensing images taken at different times.

[0086] (10) An apparatus for detecting abnormal objects using a neural network, comprising:

[0087] A memory that stores computer programs; and

[0088] A processor configured to perform the following operations by executing the computer program:

[0089] Calculate a difference image between an image captured for a given scene and a reference image, wherein the difference image indicates pixel-level differences between the captured image and the reference image;

[0090] The first encoder of the neural network extracts multiple feature maps of multiple sizes for the reference image;

[0091] The second encoder of the neural network extracts multiple feature maps of the multiple sizes from the captured image;

[0092] The third encoder of the neural network extracts multiple feature maps of the multiple sizes for the difference image;

[0093] The feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder are fused together, and the fused feature map is input into the decoder of the neural network.

[0094] The decoder generates a transformation map of the same size as the captured image based on the fused feature map, wherein the value of each pixel in the transformation map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image; and

[0095] The abnormal objects appearing in the scene are identified based on the change graph.

[0096] (11) The apparatus according to (10), wherein the reference image is a pre-captured image of the scene excluding abnormal objects, and the reference image is registered with the captured image.

[0097] (12) The apparatus according to (10), wherein the change map is a binary map, wherein a pixel value of 1 in the binary map indicates that a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image, and a pixel value of 0 indicates that a pixel in the captured image has not undergone a semantic change relative to a corresponding pixel in the reference image.

[0098] (13) According to the apparatus of (12), the processor is further configured to:

[0099] One or more detectors for detecting one or more known objects are applied to the captured image to detect known objects in the captured image;

[0100] Based on the detection results, the values ​​of pixels corresponding to the detected known objects are set to 0 in the change map; and

[0101] The updated change graph is used to identify anomalous objects in the scene.

[0102] (14) According to the apparatus of (10), the processor is further configured to:

[0103] The first encoder, the second encoder, and the third encoder respectively extract N feature maps of N sizes for the corresponding image, wherein the sizes of the N feature maps decrease in order, and N is a natural number greater than 2;

[0104] The feature maps of the same size extracted by the first encoder, the second encoder and the third encoder are fused together, and the resulting N fused feature maps are input into the decoder.

[0105] The decoder performs N-1 upsampling operations based on the received N fused feature maps.

[0106] In the first upsampling, the decoder performs upsampling on the Nth fused feature map with the smallest received size;

[0107] In each of the second to N-1th upsampling operations, the decoder performs upsampling on the feature map that has been generated by splicing the following feature maps: the previous feature map obtained from the previous upsampling operation, and a fused feature map of the same size as the previous feature map.

[0108] (15) The apparatus according to (10), wherein the neural network is trained using a training image set based on a cross-entropy loss function, wherein the training image set includes mutually registered captured image training samples and reference image training samples, and corresponding labeled variation maps.

[0109] The processor is also configured to use a trained neural network to detect the anomalous object.

[0110] (16) A storage medium storing a computer program, which, when executed by a computer, causes the computer to perform a method for detecting anomalies according to any one of (1)-(9).

Claims

1. A method for detecting abnormal objects using a neural network, comprising: To capture an image of a scene; Calculate a difference image between a captured image and a reference image, wherein the difference image indicates pixel-level differences between the captured image and the reference image; The first encoder of the neural network extracts multiple feature maps of multiple sizes for the reference image, the second encoder of the neural network extracts multiple feature maps of the multiple sizes for the captured image, and the third encoder of the neural network extracts multiple feature maps of the multiple sizes for the difference image. The feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder are fused together, and the fused feature map is input into the decoder of the neural network. The decoder generates a change map of the same size as the captured image based on the fused feature map, wherein the value of each pixel in the change map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image, the semantic change referring to a change in the pixel's category; and Based on the change map, abnormal objects appearing in the scene are identified, wherein the regions in the captured image that correspond to the regions in the change map that show semantic changes are determined as the abnormal objects.

2. The method according to claim 1, wherein, The reference image is a pre-captured image of the scene excluding any abnormal objects, and the reference image is registered with the captured image.

3. The method according to claim 1, wherein, The change image is a binary image. In the binary image, a pixel value of 1 indicates that a pixel in the captured image has undergone a semantic change relative to the corresponding pixel in the reference image, and a pixel value of 0 indicates that a pixel in the captured image has not undergone a semantic change relative to the corresponding pixel in the reference image.

4. The method according to claim 3, further comprising: One or more detectors for detecting one or more known objects are applied to the captured image to detect known objects in the captured image; Based on the detection results, the values ​​of the pixels corresponding to the detected known objects are set to 0 in the change map; as well as The updated change graph is used to identify anomalous objects in the scene.

5. The method according to claim 1, further comprising: The first encoder, the second encoder, and the third encoder respectively extract N feature maps of N sizes for the corresponding image, wherein the sizes of the N feature maps decrease in order, and N is a natural number greater than 2; The feature maps of the same size extracted by the first encoder, the second encoder and the third encoder are fused together, and the resulting N fused feature maps are input into the decoder. The decoder performs N-1 upsampling operations based on the received N fused feature maps. In the first upsampling, the decoder performs upsampling on the Nth fused feature map with the smallest received size; In each of the second to N-1th upsampling operations, the decoder performs upsampling on the feature map that has been generated by splicing the following feature maps: the previous feature map obtained from the previous upsampling operation, and a fused feature map of the same size as the previous feature map.

6. The method according to claim 1, further comprising: The neural network is trained using a training image set based on a cross-entropy loss function, wherein the training image set includes mutually registered training images and reference training images, and corresponding labeled change maps; and The abnormal object is detected using a trained neural network.

7. The method according to claim 1, wherein, The scenario refers to the location or area that needs to be monitored.

8. The method according to claim 1, wherein, The reference image and the captured image are images of the scene captured by a fixed camera device at different times, or The reference image and the captured image are remote sensing images taken at different times.

9. An apparatus for detecting abnormal objects using a neural network, comprising: A memory that stores computer programs; as well as A processor configured to perform the following operations by executing the computer program: Calculate the difference image between a captured image and a reference image for a given scene, wherein the difference image indicates the pixel-level difference between the captured image and the reference image; The first encoder of the neural network extracts multiple feature maps of multiple sizes for the reference image; The second encoder of the neural network extracts multiple feature maps of the multiple sizes from the captured image; The third encoder of the neural network extracts multiple feature maps of the multiple sizes for the difference image; The feature maps of the same size extracted by the first encoder, the second encoder, and the third encoder are fused together, and the fused feature map is input into the decoder of the neural network. The decoder generates a transformation map of the same size as the captured image based on the fused feature map, wherein the value of each pixel in the transformation map indicates whether a pixel in the captured image has undergone a semantic change relative to a corresponding pixel in the reference image, the semantic change referring to a change in the pixel's category; and Based on the change map, abnormal objects appearing in the scene are identified, wherein the regions in the captured image that correspond to the regions in the change map that show semantic changes are determined as the abnormal objects.

10. A storage medium storing a computer program, which, when executed by a computer, causes the computer to perform a method for detecting anomalies according to any one of claims 1-8.

Images