Target detection method and device, storage medium and electronic device

Abstract

The present disclosure relates to a target detection method, device, storage medium and electronic equipment, the method comprising: acquiring a first image and a second image, the second image having the same background as the first image and not having an abnormal target; fusing the first image and the second image to obtain a fused semantic image; and performing target detection on the fused semantic image to obtain an abnormal target in the first image. The present disclosure can add background image information to the fused semantic image, thereby achieving the effect of information enhancement. Performing target detection on the fused semantic image can significantly improve the detection rate and accuracy of the abnormal target, and make the target detection method have better robustness.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to a target detection method and device, a storage medium and an electronic device. BACKGROUND

[0002] Unknown object intrusion detection is an important problem in the field of computer vision and intelligent video surveillance. Unknown object detection is crucial for machine learning applications centered on security, such as abnormal object detection in medical images, abnormal object detection in autonomous driving applications, etc.

[0003] However, abnormal object detection is very complex. Unlike traditional target detection and semantic segmentation tasks, when detecting unknown object intrusion, nothing is known about the texture and shape of the abnormal object that appears, which causes difficulties in detecting abnormal objects in a scene using machine learning methods. SUMMARY

[0004] To solve at least one of the above technical problems, the present disclosure provides a technical solution for target detection.

[0005] According to some embodiments of the present disclosure, a target detection method is provided, which includes: acquiring a first image and a second image, the second image being an image with the same background as the first image and without an abnormal target; fusing the first image and the second image to obtain a fused semantic image; and performing target detection on the fused semantic image to obtain an abnormal target in the first image. Based on the above configuration, background image information can be added to the fused semantic image, thereby achieving the effect of information enhancement. Performing target detection on this fused semantic image can significantly improve the detection rate and accuracy of abnormal targets, and make the target detection method have better robustness.

[0006] In some possible implementations, the number of channels of a position in the fused semantic image corresponding to a first position is the same as the number of channels of the first position in the first image, and the first position is any position in the first image. Based on the above configuration, by limiting the number of channels of the fused semantic image, it can be ensured that the image represented by the fused semantic image conforms to the natural distribution of images, so that the method of performing target detection on natural images can also be applied to processing the fused semantic image, thereby reducing the difficulty of performing target detection on the fused semantic image.

[0007] In some possible implementation manners, the fusing the first image and the second image to obtain a fused semantic image comprises: performing pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image. Based on the above configuration, the fused semantic image conforming to a natural distribution can be quickly obtained, the efficiency of target detection based on the fused semantic image is improved, and the difficulty of target detection is reduced.

[0008] In some possible implementation manners, the performing pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image comprises: performing pixel-by-pixel pixel information splicing on the first image and the second image to obtain a spliced image; and performing feature extraction on the spliced image to obtain the fused semantic image. Based on the above configuration, the pixel information of the pixel points with a corresponding relationship in the first image and the second image can be quickly fused through simple splicing, so that each pixel in the fused semantic image has the same number of channels as the pixel at a corresponding position in the first image, and the pixel information in the first image and the second image is largely preserved, and a high-quality fused semantic image conforming to a natural distribution is obtained.

[0009] In some possible implementation manners, the performing target detection on the fused semantic image to obtain the abnormal target in the first image comprises: performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image, the mask information being used to identify the abnormal target in the first image; and determining the abnormal target in the first image according to the mask information. Based on the above configuration, the mask information corresponding to the abnormal target in the fused semantic image can be segmented based on the semantic segmentation manner, so that the abnormal target in the first image is accurately determined.

[0010] In some possible implementation manners, the performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image comprises: performing encoding processing on the fused semantic image to obtain first feature information; and performing decoding processing on the first feature information to obtain the mask information. Based on the above configuration, the fused semantic image can be encoded and decoded, so that the abnormal target in the fused semantic image is accurately segmented.

[0011] In some possible implementation manners, the first feature information comprises second feature information and third feature information, and the encoding processing on the fused semantic image to obtain the first feature information comprises: performing feature extraction on the fused semantic image to obtain the second feature information; performing feature extraction on the second feature information for at least one layer to obtain at least one fourth feature information; and fusing the at least one fourth feature information to obtain the third feature information. Based on the above configuration, the first feature information with higher richness can be obtained, so as to improve the target detection accuracy.

[0012] In some possible implementation manners, the decoding processing on the first feature information to obtain the mask information comprises: performing up-sampling on the third feature information to obtain fifth feature information; and fusing the second feature information and the fifth feature information to obtain the mask information. Based on the above configuration, the first feature information can be sufficiently decoded, so as to obtain the mask information for representing the abnormal target.

[0013] According to some other embodiments of the present disclosure, a target detection apparatus is provided, which comprises: an image acquisition module, configured to acquire a first image and a second image, the second image being an image with the same background as the first image and without an abnormal target; a fusion module, configured to fuse the first image and the second image to obtain a fused semantic image; and a detection module, configured to perform target detection on the fused semantic image to obtain an abnormal target in the first image.

[0014] In some possible implementation manners, a number of channels of a position corresponding to a first position in the fused semantic image is the same as a number of channels of the first position in the first image, and the first position is any position in the first image.

[0015] In some possible implementation manners, the fusion module is configured to perform the following operation: performing pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image.

[0016] In some possible implementation manners, the fusion module is configured to perform the following operation: performing pixel-by-pixel pixel information splicing on the first image and the second image to obtain a spliced image; and performing feature extraction on the spliced image to obtain the fused semantic image.

[0017] In some possible implementation manners, the detection module is configured to perform the following operation: performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image, the mask information being used to identify an abnormal target in the first image; and determining the abnormal target in the first image according to the mask information.

[0018] In some possible implementation manners, the detection module is configured to perform encoding processing on the fused semantic image to obtain first feature information, and perform decoding processing on the first feature information to obtain the mask information.

[0019] In some possible implementation manners, the detection module is configured to perform feature extraction on the fused semantic image to obtain the second feature information, perform at least one layer of feature extraction on the second feature information to obtain at least one fourth feature information, and fuse the at least one fourth feature information to obtain the third feature information.

[0020] In some possible implementation manners, the detection module is configured to perform up-sampling on the third feature information to obtain fifth feature information, and fuse the second feature information and the fifth feature information to obtain the mask information.

[0021] According to another embodiment of the present disclosure, an electronic device is also provided, which includes at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements the target detection method according to any one of the above embodiments by executing the instructions stored in the memory.

[0022] According to another embodiment of the present disclosure, a computer readable storage medium is also provided, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the target detection method according to any one of the above embodiments.

[0023] According to another embodiment of the present disclosure, a computer program or instructions is also provided, which is executed by a processor to implement the target detection method according to any one of the above embodiments.

[0024] In the embodiments of the present disclosure, background image information can be added in the fused semantic image, so as to achieve the effect of information enhancement. The target detection on the fused semantic image can significantly improve the detection rate and accuracy of abnormal targets, and make the target detection method have better robustness.

[0025] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure.

[0026] Other features and aspects of the present disclosure will become apparent from the following detailed description of example embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art and the advantages thereof, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0028] Figure 1 A flow chart of a target detection method according to an embodiment of the present disclosure is shown;

[0029] Figure 2 A flow chart of a specific acquisition method of a fused semantic image according to an embodiment of the present disclosure is shown;

[0030] Figure 3 A flow chart of a fused pixel information acquisition method according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A codec schematic diagram according to an embodiment of the present disclosure is shown;

[0032] Figure 5 A target detection method architecture diagram according to an embodiment of the present disclosure is shown;

[0033] Figure 6 A block diagram of a target detection device according to an embodiment of the present disclosure is shown;

[0034] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0035] Figure 8 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present specification will be described clearly and completely in the following with reference to the drawings in the embodiments of the present specification. Obviously, the described embodiments are only some embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present disclosure.

[0037] It is to be understood that the terms "first", "second", and the like, used in the description and the claims of the present application as well as the above description of the drawings merely refer to structure that is different than other structure, and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that the use of the term "or" in the examples is used to mean "and / or", that the indefinite articles "a" and "an" are used to mean "one or more" or "at least one" and the indefinite article "the" is not limiting, that the use of "including" means "comprising" or "consisting of" and visit versa, and that the use of "based on" simply means "based, at least in part, on".

[0038] Various exemplary embodiments, features, and aspects of the present disclosure will be described herein below with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0039] The term "exemplary" is used herein in the sense of serving as an example, instance, or illustration. Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0040] The term "and / or" in this document is used to mean "and" or "or", that is, "and / or" will be interpreted to mean either "solely A" or "both A and B" or "neither A nor B" or "only one of A or B". In addition, the term "at least one of' in this document is used to mean "any one or more of' or "any combination of one or more of' for example, the phrase "at least one of A, B, and C" is interpreted to mean "A or B or C or any combination of these".

[0041] In addition, numerous specific details are set forth in the following detailed description justifying better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure can be practiced without certain specific details, which are not essential for the understanding of the present disclosure. In some instances, well-known methods, structures, elements, and circuits have not been described in detail in order to avoid obscuring the subject matter of the present disclosure.

[0042] Unknown object intrusion detection is an important problem in the field of computer vision and intelligent video surveillance. Unknown object detection is crucial for machine learning applications centered on security, such as abnormal object detection in medical images, abnormal object detection in autonomous driving applications, etc. However, abnormal object detection is very complex, unlike traditional object detection and semantic segmentation tasks. In unknown object intrusion detection, nothing is known about the texture and shape of the abnormal object that may appear, which causes difficulties in detecting the abnormal object in the scene using machine learning methods.

[0043] The shapes and colors of anomalous objects appearing in different scenes can be highly diverse, making it difficult for object detection methods in related technologies to capture such a wide variety of anomalous objects. Furthermore, changing non-anomalous objects in the scene can easily interfere with object detection algorithms, leading to false positives and false negatives. This is a major reason why the accuracy of anomalous object detection in related technologies is relatively low. Moreover, related technologies typically focus on anomalous object detection in outdoor scenes, with relatively little research and data available for indoor scenes. This results in a lack of ability for object detection methods in related technologies to perceive anomalous objects in indoor scenes.

[0044] To address at least one of the aforementioned technical problems, embodiments of this disclosure provide a target detection method. This method can obtain a fused semantic image that conforms to the natural image distribution through image fusion. Compared to the images directly obtained in related technologies, this fused semantic image adds background image information. Detecting abnormal objects based on the fused semantic image can significantly improve the accuracy of abnormal object detection, making the abnormal object detection results less affected by moving normal objects, thus improving the robustness of abnormal object detection.

[0045] This disclosure does not limit the detected abnormal objects. For example, they can be real people, animals, plants, knives, workpieces, fireworks, etc. Accordingly, this disclosure can be applied to scenarios such as dangerous goods detection and foreign object intrusion detection. This disclosure does not limit the application scenarios and can be applied to various unidentified object detection scenarios. Of course, the detected abnormal objects can be regarded as intrusive unidentified objects.

[0046] The target detection method provided in this disclosure can be executed by a terminal device, a server, or other types of electronic devices. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the target detection method can be implemented by a processor calling computer-readable instructions stored in memory. The target detection method of this disclosure embodiment is described below using an electronic device as the execution subject as an example. This target detection method is implemented by a processor calling computer-readable instructions stored in memory.

[0047] Figure 1 A flowchart of a target detection method according to an embodiment of the present disclosure is shown, such as...Figure 1 The method includes the following steps:

[0048] S101: Obtain a first image and a second image, the second image being an image with the same background as the first image and without an abnormal target.

[0049] The method of obtaining the first image is not limited in the embodiments of the present disclosure, which can be an image obtained by photographing a scene to be detected, or an image frame in a video obtained by video photographing the scene. The color system of the first image is not limited in the embodiments of the present disclosure, for example, the first image can be a grayscale image, a black-and-white image or a color image. The resolution of the first image is not limited in the embodiments of the present disclosure, for example, the resolution can be 6576x4384 or 1920x1300. The first image can be an image obtained directly by photographing, or an image obtained by preprocessing the image obtained by photographing. The preprocessing method is not limited in the embodiments of the present disclosure, for example, it can be adjusting contrast, adjusting brightness, adjusting color system, noise reduction, sharpening, etc. The second image is a background image of the scene without an abnormal object, and the obtaining method can be the same as that of the first image, which is not described herein. The second image does not have an abnormal target, but can have a non-abnormal target, for example, there can be a normal target in motion.

[0050] There are few solutions for indoor anomaly detection in the related art, which are not suitable for indoor scene anomaly object detection. Therefore, in order to be suitable for indoor scene anomaly object detection, the embodiments of the present disclosure can photograph an indoor scene to obtain the first image and the second image. By image extraction of the indoor scene and fusion and target detection processing of the images in subsequent operations, the indoor scene anomaly object can be analyzed, so that the embodiments of the present disclosure can be applied to indoor anomaly object detection scenes.

[0051] S102: Fuse the first image and the second image to obtain a fused semantic image.

[0052] The first image and the second image can be preprocessed before fusion in the embodiments of the present disclosure, so that the first image and the second image meet the requirements of the fusion operation. For example, if the fusion operation requires the images to be fused to have the same number of channels, the same size or the same resolution, the first image and the second image can be adjusted accordingly so that they have the same number of channels, the same size or the same resolution. The preprocessing in the embodiments of the present disclosure is not limited, and is mainly to meet the fusion requirements. By fusing the first image and the second image, the fused semantic image can carry background image information, thereby improving the accuracy of abnormal object detection. Moreover, the second image can also include a moving normal target. Based on the fused semantic image obtained from such a second image, the interference formed by the normal target can be avoided, thereby significantly enhancing the robustness and improving the accuracy of target detection.

[0053] In one embodiment, the number of channels of the position corresponding to the first position in the fused semantic image is the same as the number of channels of the first position in the first image, and the first position is any position in the first image. By limiting the number of channels of the fused semantic image, it can be ensured that the image represented by the fused semantic image conforms to the natural distribution of images, so that the method of performing target detection on natural images can also be applied to processing the fused semantic image, thereby reducing the difficulty of target detection on the fused semantic image.

[0054] For example, a natural image has three channels of RGB, which represent the Red, Green and Blue values of the pixels in the image, respectively. When the natural image is input into a certain target detection model, the corresponding target detection result can be obtained. The fused semantic image also has three channels, that is, the fused semantic image has the same representation as the natural image and can be represented by the same data structure. Therefore, the above-mentioned certain target detection model can also be applied to target detection on the fused semantic image without any adaptive adjustment of the target detection model.

[0055] In one embodiment, the fusion of the first image and the second image includes pixel-by-pixel alignment fusion of the first image and the second image to obtain the fused semantic image. By using the alignment fusion method, a fused semantic image conforming to the natural distribution can be quickly obtained, thereby improving the efficiency of target detection based on the fused semantic image and reducing the difficulty of target detection.

[0056] The embodiments of the present disclosure do not limit the specific fusion method used for the alignment fusion, for example, in an embodiment, the first image and the second image can be spliced pixel by pixel to obtain a spliced image; and the spliced image is subjected to feature extraction to obtain the fusion semantic image. This fusion method can quickly fuse the pixel information of the pixel points with the corresponding relationship in the first image and the second image through simple splicing, so that each pixel in the fusion semantic image has the same number of channels as the pixel at the corresponding position in the first image, and the pixel information in the first image and the second image is largely preserved to obtain a high-quality fusion semantic image conforming to the natural distribution.

[0057] In a specific embodiment, a specific acquisition method of the fusion semantic image is provided, as shown in the following table. Figure 2 The method includes the following steps.

[0058] S1021: The first pixel information and the second pixel information are fused to obtain fusion pixel information corresponding to the first position, the first pixel information representing the pixel information of the first position in the first image, the second pixel information representing the pixel information of the position corresponding to the first position in the second image, and the first position being any position in the first image.

[0059] The embodiments of the present disclosure do not limit the fusion method of the first pixel information and the second pixel information, for example, direct addition, weighting, normalization, convolution, etc. can be performed, and the fusion pixel information containing the background image information can be obtained through the fusion. For example, as shown in the following table, Figure 3 The fusion of the first pixel information and the second pixel information to obtain the fusion pixel information corresponding to the first position can include the following steps.

[0060] S1: The first pixel information and the second pixel information are spliced to obtain spliced pixel information corresponding to the first position.

[0061] Taking the first pixel information A1 of the first image A as the pixel information corresponding to the indoor P position and the second pixel information B1 of the second image B as the pixel information corresponding to the indoor P position as an example, A1 includes RGB three values {r1, g1, b1}, and B1 also includes RGB three values {r2, g2, b2}. The spliced pixel information can be represented as {r1, g1, b1, r2, g2, b2}, that is, if the first image has three channels and is represented by a matrix with three channels, the spliced pixel information corresponding to each pixel point in the first image can form a matrix with six channels.

[0062] S2: performing feature extraction on the spliced pixel information corresponding to the first position to obtain fusion pixel information corresponding to the first position.

[0063] The feature extraction is performed on the matrix with six channels to obtain a matrix with three channels, which has the same size as the matrix corresponding to the first image. Based on the configuration, the fusion pixel information corresponding to each pixel point can be obtained, and the fusion pixel information has the same number of channels as the pixel information corresponding to the pixel point in the first image, so that the fusion semantic image conforming to the natural distribution can be obtained based on the fusion pixel information.

[0064] S1022: obtaining the fusion semantic image according to the fusion pixel information corresponding to the first position and the fusion pixel information corresponding to the second position, the second position representing other positions in the first image different from the first position.

[0065] In the embodiment of the present disclosure, the acquisition manner of the fusion pixel information corresponding to the second position is the same as that of the fusion information corresponding to the first position, and the embodiment of the present disclosure does not repeat it. In one specific embodiment, the first image and the second image can be directly input into the trained semantic image processing module to obtain the semantic image output by the module, and the semantic image is the fusion semantic image, and the semantic image processing module can execute steps S1021 to S1022.

[0066] In the process of executing step S1021, the first pixel information and the second pixel information can be spliced to obtain the spliced pixel information corresponding to the first position. The feature extraction is performed on the spliced pixel information corresponding to the first position to obtain the fusion pixel information corresponding to the first position. The embodiment of the present disclosure does not limit the specific method of feature extraction. For example, at least one convolution operation can be used for feature extraction, or a self-attention operation can be used for feature extraction. For example, a convolutional neural network with a multi-layer structure can be used for the feature extraction, each layer of the convolutional neural network includes 6 and 3 convolution filters respectively, and the convolution kernel size of each filter is 3*3. By inputting the first image and the second image into the semantic image, the semantic image can obtain the fusion semantic image conforming to the natural picture distribution by using the convolutional neural network with the multi-layer structure. Based on the configuration, the fusion semantic image conforming to the natural distribution can be obtained, thereby reducing the difficulty of detecting the fusion semantic image.

[0067] S103: performing target detection on the fusion semantic image to obtain an abnormal target in the first image.

[0068] Of course, the embodiments of the present disclosure do not limit the specific method of target detection, for example, the above target detection can be performed using a one-stage target detection method, a two-stage target detection method or a coding method. Based on the above configuration, background image information can be added to the fused semantic image, thereby achieving the effect of information enhancement. Target detection can be performed on the fused semantic image, which can significantly improve the detection rate and accuracy of abnormal targets, and make the target detection method have better robustness.

[0069] In one embodiment, the above target detection on the fused semantic image to obtain the abnormal target in the first image includes: performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image, the mask information being used to identify the abnormal target in the first image; and determining the abnormal target in the first image according to the mask information. The region representing the abnormal target in the semantic image can be segmented by the semantic segmentation method, and the corresponding mask information can be generated according to the position of the region. Since the semantic image is obtained by aligning and fusing the first image and the second image, the mask information can also represent the position of the abnormal target in the first image. The abnormal target in the first image can be determined based on the mask information and the first image. Based on the above configuration, the mask information corresponding to the abnormal target can be segmented in the fused semantic image based on the semantic segmentation method, so as to accurately determine the abnormal target in the first image.

[0070] In one embodiment, the above fused semantic image can be encoded to obtain first feature information. The first feature information is decoded to obtain mask information, and the mask information is used to identify the abnormal target in the first image. In one embodiment, the mask information can form a mask image with the same size as the first image. The embodiments of the present disclosure do not limit the method of encoding processing, for example, the convolutional neural network (CNN) or deep convolutional neural network (DCNN) can be used. Based on the above configuration, the fused semantic image can be encoded and decoded, so as to accurately segment the abnormal target in the fused semantic image.

[0071] In an embodiment, the first feature information can include second feature information and third feature information, and the encoding of the fused semantic image to obtain the first feature information includes: performing feature extraction on the fused semantic image to obtain the second feature information. At least one layer of feature extraction is performed on the second feature information to obtain at least one fourth feature information. The at least one fourth feature information is fused to obtain the third feature information. Of course, the specific operations of feature extraction and fusion are not limited in the embodiments of the present disclosure, and can be referred to the foregoing. Based on the foregoing configuration, the first feature information with high richness can be obtained, so as to improve the target detection accuracy.

[0072] Please refer to Figure 4 , which shows a codec structure diagram. In the Encoder (encoder) link, feature extraction can be performed on the fused semantic image to obtain second feature information T2. In Figure 4 , an Atrous Conv (dilated convolution) is used for feature extraction. After the second feature information T2 passes through five network structures respectively, five fourth feature information (T4) are obtained, and the five network structures are respectively a convolution layer with a convolution kernel of 1*1, a convolution layer with a convolution kernel of 3*3 and a learning rate of 6, a convolution layer with a convolution kernel of 3*3 and a learning rate of 12, a convolution layer with a convolution kernel of 3*3 and a learning rate of 18, and a pooling layer. The five fourth feature information (T4) are fused to obtain third feature information (T3). The fusion can be realized by a convolution layer with a convolution kernel of 1*1.

[0073] In an embodiment, the decoding of the first feature information to obtain the mask information includes: up-sampling the third feature information to obtain fifth feature information. The second feature information and the fifth feature information are fused to obtain the mask information. Of course, the specific operations of up-sampling and fusion are not limited in the embodiments of the present disclosure, and can be referred to the foregoing. Based on the foregoing configuration, the first feature information can be fully decoded to obtain the mask information for representing the abnormal target.

[0074] Please refer to Figure 4 , in the Decoder (decoder) link, the second feature information T2 can be directly input into the decoder, the third feature information (T3) is up-sampled to obtain fifth feature information (T5), the convolution operation is performed on the second feature information (T2), and the convolution result is connected with the fifth feature information (T5) to realize preliminary fusion. Then, the preliminary fusion result is processed by a convolution layer with a convolution kernel of 3*3 and is up-sampled, and the mask information can be obtained.

[0075] In some embodiments, step S103 can be performed by inputting the fused semantic image into the target detection model to obtain the abnormal target in the first image. The target detection model is obtained by adjusting the parameters of the pre-trained network. In this embodiment, the fused semantic image has a natural distribution characteristic, meaning it is compatible with target detection networks in related technologies. These target detection networks can be directly used as pre-trained networks, and their parameters can be adjusted to process the fused semantic image in step S103, thereby outputting the abnormal target. Since the target detection networks in related technologies have undergone extensive training and possess the ability to perceive regular objects, obtaining the target detection model by fine-tuning the parameters of these target detection networks can reduce the dependence of the target detection model on annotations and samples, thus reducing the training cost of the target detection model. Compared to related technologies that mainly rely on optical flow, saliency maps, etc., for unidentified object intrusion detection, lacking large-scale available pre-trained models and thus relying on a large number of labeled samples for training, this embodiment clearly has significant advantages.

[0076] This disclosure provides a target detection method, such as... Figure 5 As shown, a fused semantic image (semantic map) conforming to a natural distribution is obtained by fusing the background image. Inputting this semantic map into a target detection network can identify abnormal targets. Specifically, the target detection network can include an encoder-decoder module. The encoder module receives the semantic map as input, extracts features from the semantic map using a multi-layer convolutional neural network and a porous spatial pyramid pooling module, and outputs multi-scale features to the decoder for abnormal object region segmentation. The decoder receives the multi-scale features output by the encoder as input, concatenates the low-level features with the high-level features after convolution, and then passes them through a 3*3 convolutional layer and upsamples them to the same size as the semantic map input to the encoder, thereby obtaining the binary mask of the abnormal object in the input image of the semantic map encoding module. Based on the above configuration, this embodiment of the present disclosure implements abnormal object detection based on neural networks. Compared with related technologies using optical flow or saliency map detection, this embodiment of the present disclosure can obtain more accurate detection results and has a significant advantage in robustness against interference from moving normal objects.

[0077] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0078] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further.

[0079] In addition, the disclosure also provides a target detection device, an electronic device, a computer readable storage medium, and a program, which can be used to implement any one of the target detection methods provided by the disclosure. The corresponding technical solutions and descriptions are described in the method part and are not repeated here.

[0080] Figure 6 An apparatus for target detection is shown according to an embodiment of the disclosure. As shown in Figure 6 The apparatus comprises:

[0081] An image acquisition module 101 is configured to acquire a first image and a second image, wherein the second image has the same background as the first image and does not contain an abnormal target.

[0082] A fusion module 102 is configured to fuse the first image and the second image to obtain a fused semantic image.

[0083] A detection module 103 is configured to perform target detection on the fused semantic image to obtain an abnormal target in the first image.

[0084] In some possible implementation manners, the number of channels of a position corresponding to a first position in the fused semantic image is the same as the number of channels of the first position in the first image, and the first position is any position in the first image.

[0085] In some possible implementation manners, the fusion module is configured to perform pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image.

[0086] In some possible implementation manners, the fusion module is configured to perform pixel-by-pixel pixel information splicing on the first image and the second image to obtain a spliced image, and perform feature extraction on the spliced image to obtain the fused semantic image.

[0087] In some possible implementation manners, the detection module is configured to perform semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image, wherein the mask information is used to identify the abnormal target in the first image, and determine the abnormal target in the first image according to the mask information.

[0088] In some possible implementation manners, the detection module is configured to perform encoding processing on the fused semantic image to obtain first feature information, and perform decoding processing on the first feature information to obtain the mask information.

[0089] In some possible implementation manners, the detection module is configured to perform the following operation: performing feature extraction on the fused semantic image to obtain the second feature information; performing feature extraction on the second feature information for at least one layer to obtain at least one fourth feature information; and performing fusion on the at least one fourth feature information to obtain the third feature information.

[0090] In some possible implementation manners, the detection module is configured to perform the following operation: performing up-sampling on the third feature information to obtain fifth feature information; and performing fusion on the second feature information and the fifth feature information to obtain the mask information.

[0091] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can be referred to the description of the above method embodiments. For briefness, details are not described herein.

[0092] The embodiments of the present disclosure also provide a computer-readable storage medium, and the computer-readable storage medium stores at least one instruction or at least one program. The at least one instruction or at least one program is loaded and executed by a processor to implement the above method. The computer-readable storage medium can be a nonvolatile computer-readable storage medium.

[0093] Specifically, the method at least includes the following contents:

[0094] According to some embodiments of the present disclosure, a target detection method is provided, which includes: acquiring a first image and a second image, the second image having the same background as the first image and not having an abnormal target; fusing the first image and the second image to obtain a fused semantic image; and performing target detection on the fused semantic image to obtain an abnormal target in the first image. Based on the above configuration, background image information can be added to the fused semantic image, so as to achieve the effect of information enhancement. Performing target detection on the fused semantic image can significantly improve the detection rate and accuracy of the abnormal target, and make the target detection method have better robustness.

[0095] In some possible implementation manners, a number of channels of a position corresponding to a first position in the fused semantic image is the same as a number of channels of the first position in the first image, and the first position is any position in the first image. Based on the above configuration, by limiting the number of channels of the fused semantic image, it can be ensured that the image represented by the fused semantic image conforms to the natural distribution of images, so that the method of performing target detection on natural images can also be applied to processing the fused semantic image, thereby reducing the difficulty of performing target detection on the fused semantic image.

[0096] In some possible implementation manners, the fusing the first image and the second image to obtain the fused semantic image comprises: performing pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image. Based on the configuration, the fused semantic image conforming to the natural distribution can be quickly obtained, the efficiency of target detection based on the fused semantic image is improved, and the difficulty of target detection is reduced.

[0097] In some possible implementation manners, the performing pixel-by-pixel alignment fusion on the first image and the second image to obtain the fused semantic image comprises: performing pixel information splicing on the first image and the second image to obtain a spliced image; and performing feature extraction on the spliced image to obtain the fused semantic image. Based on the configuration, the pixel information of the pixel points with the corresponding relationship in the first image and the second image can be quickly fused through simple splicing, so that each pixel in the fused semantic image has the same number of channels as the pixel at the corresponding position in the first image, and the pixel information in the first image and the second image is largely preserved, and a high-quality fused semantic image conforming to the natural distribution is obtained.

[0098] In some possible implementation manners, the performing target detection on the fused semantic image to obtain the abnormal target in the first image comprises: performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image, the mask information being used to identify the abnormal target in the first image; and determining the abnormal target in the first image according to the mask information. Based on the configuration, the mask information corresponding to the abnormal target can be segmented from the fused semantic image in a semantic segmentation manner, so that the abnormal target in the first image is accurately determined.

[0099] In some possible implementation manners, the performing semantic-based segmentation on the fused semantic image to obtain mask information corresponding to the fused semantic image comprises: performing encoding processing on the fused semantic image to obtain first feature information; and performing decoding processing on the first feature information to obtain the mask information. Based on the configuration, the fused semantic image can be encoded and decoded, so that the abnormal target is accurately segmented from the fused semantic image.

[0100] In some possible implementation, the first feature information includes second feature information and third feature information, and the encoding of the fused semantic image to obtain the first feature information includes: performing feature extraction on the fused semantic image to obtain the second feature information; performing feature extraction on the second feature information for at least one layer to obtain at least one fourth feature information; and fusing the at least one fourth feature information to obtain the third feature information. Based on the configuration, the first feature information with higher richness can be obtained, so as to improve the target detection accuracy.

[0101] In some possible implementation, the decoding of the first feature information to obtain the mask information includes: up-sampling the third feature information to obtain fifth feature information; and fusing the second feature information and the fifth feature information to obtain the mask information. Based on the configuration, the first feature information can be fully decoded, so as to obtain the mask information for representing the abnormal target.

[0102] According to some other embodiments of the present disclosure, a computer program or instructions are also provided, which, when executed by a processor, implement the above-mentioned target detection method.

[0103] The embodiments of the present disclosure also provide an electronic device, which includes: a processor; a memory for storing processor-executable instructions; and wherein the processor is configured to implement the above-mentioned method.

[0104] The electronic device can be provided as a terminal, a server, or other forms of devices.

[0105] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. For example, the electronic device 800 can be a terminal such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

[0106] Referring to Figure 7 The electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0107] The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete the steps of the methods described above, in whole or in part. Moreover, the processing component 802 can include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0108] The memory 804 is configured to store various types of data to support the operations of the electronic device 800. Examples of these data include instructions to operate any applications or methods on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be realized by any type of volatile or non-volatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0109] The power component 806 provides power to the various components of the electronic device 800. The power component 806 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0110] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a back camera. The front camera and / or the back camera can receive external multimedia data when the electronic device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front camera and the back camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0111] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0112] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0113] The sensor component 814 includes one or more sensors for providing status assessments for various aspects of the electronic device 800. For example, the sensor component 814 can detect an open / closed position of the electronic device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change of position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, orientation or acceleration / deceleration / g-force and temperature of the electronic device 800. The sensor component 814 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0114] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast-related information from an external broadcasting management system via a broadcast channel. In an example embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technology.

[0115] In exemplary embodiments, the electronic device 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0116] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 804 including computer program instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods.

[0117] Figure 8 A block diagram of another electronic device according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server. Referring to Figure 8 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions, such as application programs, executable by the processing component 1922. The application programs stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described methods.

[0118] The electronic device 1900 can also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, or the like.

[0119] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, is also provided, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above-described methods.

[0120] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0121] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0122] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0123] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0124] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0125] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the instructions which operate on the computer or other programmable data processing apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0126] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0127] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0128] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive. Many modifications and variations of the described embodiments are possible and are within the scope of the disclosure. The selection of terms is intended to best describe the principles of the embodiments, practical application, or technical improvements in the art, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A target detection method, characterized in that, The method includes: Acquire a first image and a second image, wherein the second image has the same background as the first image and does not contain any abnormal targets; The first image and the second image are fused pixel by pixel to obtain the fused semantic image; Target detection is performed on the fused semantic image to obtain abnormal targets in the first image; The step of performing target detection on the fused semantic image to obtain abnormal targets in the first image includes: The fused semantic image is encoded to obtain first feature information, which includes second and third feature information. This step includes feature extraction from the fused semantic image to obtain the second feature information; feature extraction of the second feature information at least once to obtain at least one fourth feature information; fusion of the at least one fourth feature information to obtain the third feature information; upsampling of the third feature information to obtain fifth feature information; and fusion of the second and fifth feature information to obtain mask information, which is used to identify abnormal targets in the first image. The abnormal target in the first image is determined based on the mask information.

2. The method according to claim 1, characterized in that, The number of channels at the position corresponding to the first position in the fused semantic image is the same as the number of channels at the first position in the first image, where the first position is any position in the first image.

3. The method according to claim 1, characterized in that, The step of performing pixel-by-pixel alignment and fusion of the first image and the second image to obtain the fused semantic image includes: The first image and the second image are stitched together pixel by pixel to obtain a stitched image; Feature extraction is performed on the stitched image to obtain the fused semantic image.

4. A target detection device, characterized in that, The device includes: The image acquisition module is used to acquire a first image and a second image, wherein the second image is an image with the same background as the first image and without any abnormal targets; The fusion module is used to perform pixel-by-pixel alignment fusion of the first image and the second image to obtain the fused semantic image; The detection module is used to perform target detection on the fused semantic image to obtain abnormal targets in the first image; The step of performing target detection on the fused semantic image to obtain abnormal targets in the first image includes: The fused semantic image is encoded to obtain first feature information, which includes second and third feature information. This step includes feature extraction from the fused semantic image to obtain the second feature information; feature extraction of the second feature information at least once to obtain at least one fourth feature information; fusion of the at least one fourth feature information to obtain the third feature information; upsampling of the third feature information to obtain fifth feature information; and fusion of the second and fifth feature information to obtain mask information, which is used to identify abnormal targets in the first image. The abnormal target in the first image is determined based on the mask information.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement a target detection method as described in any one of claims 1-3.

6. An electronic device, characterized in that, The method includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor implements a target detection method as described in any one of claims 1-3 by executing the instructions stored in the memory.

Images