Target region detection method and device, storage medium and electronic device

Abstract

The application discloses a target region detection method and device, a storage medium and an electronic device. The method comprises the following steps: receiving a visible light image and an infrared image sent by a target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in a video frame obtained by the target device when image collection of the target region is performed; inputting the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region; and fusing the first detection result and a second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by performing motion detection on the infrared image of the target region, and the technical problem that the accuracy of the target region detection in a complex scene is not high in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of infrared and visible light fusion recognition, and more specifically, to a method and apparatus for detecting a target area, a storage medium, and an electronic device. Background Technology

[0002] Intelligent detection and alarm systems operating around the clock are a key focus and challenge in the security field. In low-light conditions at night or in adverse weather with low visibility, relying solely on visible light or thermal imaging images is insufficient for accurate alarm detection. To meet the demands of 24 / 7 intelligent monitoring and alarm systems, the fusion of visible light and thermal imaging images is gaining increasing attention. Currently, infrared and visible light fusion recognition can be broadly categorized into pixel-level fusion and feature-level fusion. Pixel-level fusion directly fuses the pixels of visible light and thermal imaging images, making it susceptible to noise from the images themselves, which can negatively impact the final fusion quality and recognition results. Feature-level fusion, on the other hand, involves inputting the two types of images into a convolutional neural network to extract multi-dimensional tensor features before performing feature-level fusion. Compared to pixel-level fusion, it is more lightweight and accurate, filtering out more noise but also reducing the need for detailed processing. However, this technology is highly dependent on environmental conditions and image quality. In complex scenarios such as occlusion, or when image resolution is low and noise levels are high, relying solely on the fused features can lead to low detection rates, resulting in false negatives and missed detections.

[0003] To address the aforementioned issues, existing technologies currently lack effective solutions for problems such as low accuracy in localization, detection, and classification of target areas in complex scenarios. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting target areas, a storage medium, and an electronic device, to at least solve the problems of low accuracy in positioning, detection, and classification of target areas in complex scenarios in the prior art.

[0005] According to one aspect of the present invention, a method for detecting a target region is provided, comprising: receiving a visible light image and an infrared image sent by a target device, wherein the visible light image and the infrared image are acquired from the same video frame, and the same video frame is any frame image in a video frame obtained by the target device acquiring an image of the target region; inputting the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region; fusing the first detection result with a second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target region.

[0006] According to another aspect of the present invention, a target region detection method apparatus is also provided, comprising: a receiving module, configured to receive a visible light image and an infrared image sent by a target device, wherein the visible light image and the infrared image are acquired from the same video frame, and the same video frame is any frame image in a video frame obtained by the target device in capturing images of the target region; a target detection module, configured to input the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region; and a fusion module, configured to fuse the first detection result with a second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target region.

[0007] According to another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the method in any of the method embodiments at runtime.

[0008] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to perform the method in any of the above-described method embodiments through the computer program.

[0009] In this embodiment of the invention, a target detection network is used to analyze and fuse visible light and infrared images to obtain a first detection result of the target area. This first detection result is then fused with a second detection result obtained by motion detection of the infrared image using a background model to obtain a third detection result. This process extracts and utilizes the features of both infrared and visible light images at a deeper level, enhancing the robustness and accuracy of the detection model. This achieves the goal of accurately locating, detecting, and classifying target objects within the target area, thereby improving the technical efficiency of target area detection. This allows the target device to more effectively handle all-day monitoring in various conditions, including day and night, rain, snow, fog, and obstructed scenes. This solves the technical problem of low accuracy in target area detection in complex scenarios in existing technologies. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0011] Figure 1 This is a hardware structure block diagram of a computer terminal for a target area detection method according to an embodiment of the present invention;

[0012] Figure 2 This is a flowchart of a target region detection method according to an embodiment of the present invention;

[0013] Figure 3 This is a schematic diagram of the infrared-visible light fusion deep neural network corresponding to the target detection model according to an optional embodiment of the present invention;

[0014] Figure 4 This is a flowchart illustrating a target detection model according to an optional embodiment of the present invention;

[0015] Figure 5 This is a schematic diagram of the structure of a target area detection device according to an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] The methods and embodiments provided in this application can be executed on a computer terminal, mobile terminal, or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a target area detection method according to an embodiment of the present invention. Figure 1 As shown, computer terminal 10 may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0019] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the target area detection method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0020] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0021] According to one aspect of the present invention, a method for detecting a target region is provided. Optionally, as an alternative implementation, the above-described target region detection method may be applied to, but is not limited to, [examples of other methods]. Figure 1 In the environment shown.

[0022] Optionally, in this embodiment, the terminal device can be a terminal device configured with a target client, which may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client may be a video client, instant messaging client, browser client, educational client, etc. The network may include, but is not limited to, wired network and wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI and other networks that enable wireless communication. The server may be a single server, a server cluster composed of multiple servers, or a cloud server. The above is only an example, and no limitation is made in this embodiment.

[0023] Alternatively, as an alternative implementation method, such as Figure 2 As shown, the detection method for the above-mentioned target area includes:

[0024] Step S202: Receive visible light image and infrared image sent by the target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in the video frame obtained by the target device to capture images of the target area;

[0025] Step S204: Input the visible light image and the infrared image into the target detection network to obtain the first detection result corresponding to the target region;

[0026] Step S206: The first detection result is fused with the second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target area.

[0027] Through the above steps, a target detection network is used to analyze and fuse visible light and infrared images to obtain the first detection result of the target area. This first detection result is then fused with the second detection result obtained by motion detection of the infrared image using a background model to obtain the third detection result. This process allows for deeper extraction and utilization of features from both infrared and visible light images, enhancing the robustness and accuracy of the detection model. Consequently, it achieves the goal of accurately locating, detecting, and classifying target objects within the target area, thereby improving the technical efficiency of target area detection. This enables the target device to more effectively handle all-day monitoring in various conditions, including day and night, rain, snow, fog, and obstructed scenes. This solves the technical problem of low accuracy in target area detection in complex scenarios in existing technologies.

[0028] There are multiple ways to obtain the first detection result corresponding to the target region from the visible light image and the infrared image in step S204 above. In one optional embodiment, it can be implemented by the following scheme: inputting the visible light image and the infrared image into the feature extraction and fusion module in the target detection network to obtain the fused feature map of the visible light image and the infrared image output by the feature extraction and fusion module, wherein the feature extraction and fusion module includes multiple cascaded convolutional layer modules and a tensor concatenation module; inputting the fused feature map into the hierarchical depth aggregation module in the target detection network to obtain multiple upsampled feature maps output by the hierarchical depth aggregation module; inputting the multiple upsampled feature maps into the iterative depth aggregation module in the target detection network to obtain the predicted bounding box information output by the iterative depth aggregation module, and using the predicted bounding box information as the first detection result of the target detection network for the target region.

[0029] It should be noted that, in order to ensure the correspondence between the visible light image and the infrared image input, after the visible light image and the infrared image of the target area are obtained through the target device, a certain registration and calibration is also required. Optionally, the registration and calibration judgment is as follows: the edges are used as the correlation features for the registration of the two types of images. First, the edge grayscale information is preprocessed and enhanced. The edges are extracted using the Canny operator and the affine transformation parameters are calculated using boundary correlation to achieve the registration and calibration of the two images.

[0030] Optionally, the visible light image and the infrared image are input into the feature extraction and fusion module in the target detection network to obtain a fused feature map of the visible light image and the infrared image output by the feature extraction and fusion module. This includes: extracting features from the visible light image using the multiple convolutional layer modules to obtain a visible light feature map corresponding to the visible light image; extracting features from the infrared image using the multiple convolutional layer modules to obtain an infrared feature map corresponding to the visible light image; and using the tensor stitching module to stitch the visible light feature map and the infrared feature map together to determine the fused feature map of the visible light image and the infrared image.

[0031] It is understandable that when performing image processing through the feature extraction and fusion module, the visible light image and the infrared image are processed separately. After processing, the tensor stitching module carried in the feature extraction and fusion module is used to fuse the extracted infrared feature map and the visible light feature map to obtain a fused feature map containing infrared and visible light features.

[0032] Optionally, the fused feature map is input into the hierarchical deep aggregation module in the target detection network to obtain multiple upsampled feature maps output by the hierarchical deep aggregation module, including: downsampling the fused feature map through the first-layer downsampling submodule of the hierarchical deep aggregation module to obtain a first-stage feature map corresponding to the fused feature map; wherein, the downsampling extraction is used to instruct the fused feature map to be divided into multiple basic blocks according to the preset block size of the first-layer downsampling submodule, and feature fusion is performed on each of the multiple basic blocks; the first-stage feature map is input into the hierarchical deep aggregation module. The second-layer downsampling submodule of the deep aggregation module performs downsampling extraction to obtain the second-stage feature map corresponding to the fused feature map. Then, it performs partial-layer aggregation on the second-stage feature map through a preset residual convolutional layer to determine the second target-level feature map that will be downsampled and extracted by the third-layer downsampling submodule of the hierarchical deep aggregation module. After all the downsampling submodules in the hierarchical deep aggregation module have completed downsampling extraction, each target-level feature map processed by the residual convolutional layer is upsampled to determine multiple upsampled feature maps output by the hierarchical deep aggregation module.

[0033] In short, the hierarchical deep aggregation module divides the infrared-visible light fusion feature map into multiple levels for downsampling and extraction. Within each level, it performs block feature extraction and fusion, and uses the idea of ​​skip connections. By passing the feature map of the previous level through a max pooling layer to skip to the downsampling extraction of the next level for feature fusion, the feature map of the previous level is aggregated into the current block and its size is restored by the deconvolution module. Then, according to the number of levels of the hierarchical deep aggregation module, it can output an upsampled feature map after processing the infrared-visible light fusion feature map by a number corresponding to the number of levels.

[0034] Optionally, the plurality of upsampled feature maps are input into the iterative deep aggregation module in the target detection network to obtain the predicted bounding box information output by the iterative deep aggregation module, including: performing feature fusion on the plurality of upsampled feature maps output by the hierarchical deep aggregation module through the accumulating convolution module in the iterative deep aggregation module; determining the predicted bounding box information based on the feature fusion result, wherein the predicted bounding box information includes at least one of the following: the heatmap prediction category of the predicted bounding box, the length and width prediction of the predicted bounding box, and the pixel restoration offset prediction of the predicted bounding box.

[0035] Optionally, before implementing step S206 above, the specific method for determining the second detection result includes: taking the infrared image of the first frame in the video frame acquired by the target device as the standard infrared image, and determining the background model of the target area based on the standard infrared image; judging each pixel in the infrared images of the other frames in the video frame except the first frame through the background model, so as to segment the foreground and background points in the infrared images of the other frames.

[0036] Optionally, the background model is used to determine each pixel in the infrared image of frames other than the first frame in the video frame to segment foreground and background points in the infrared image of the other frames. This includes: determining the difference between the current pixel value of each pixel in the infrared image of the other frames and the pixel value in the corresponding sample set of the background model; if the difference is within the pixel value error range corresponding to the background model, determining the current pixel as a target background point; if the difference is outside the pixel value error range corresponding to the background model, determining the current pixel as a target foreground point.

[0037] Optionally, the background model is designed as follows: V(x,y) represents the current pixel value, M(x,y)={V1(x,y),...V n (x,y) represents the historical sample set, and R is the range of the difference between the current pixel value and the historical sample value, which is set to 20 in this background model. First, the number of neighborhood sample points n in the sample set is selected as 50, which increases adaptability to complex scenes while ensuring sufficient memory usage. The threshold th is set to 43. The pixel value V(x,y) is subtracted from the points in the historical sample set M(x,y), and ±R is the allowed fluctuation range. If the number of points N>th within this range is similar to historical pixels, it is considered a background point. The judgment of foreground points is similar to that of background points. Based on the background point judgment process described above, the threshold T = n-th for judging foreground points can be set to 7. That is, if the number N1 obtained by subtracting the pixel value V(x,y) from the points in the historical sample set M(x,y) that is greater than R or less than -R is greater than T, then the pixel is judged as a foreground point.

[0038] Optionally, the infrared image of the first frame in the video frame acquired by the target device is used as a standard infrared image to determine the background model of the target area based on the standard infrared image. This includes: determining the pixel values ​​corresponding to all pixels in the standard infrared image and / or the pixel values ​​of the neighboring pixels of the pixels to obtain a sample set of the standard infrared image; and training a preset background model based on preset motion detection requirements and the sample set to obtain a background model.

[0039] Optionally, training a preset background model based on preset motion detection requirements and the sample set includes: determining the allowable pixel value error range of the preset background model based on the pixel difference and the error threshold of the preset motion detection requirements, wherein the pixel difference is the difference between the maximum and minimum pixel values ​​in the sample set; determining a target number of neighborhood sample point sets from the sample set based on the preset motion detection requirements, and obtaining a first preset threshold for determining any pixel as a background point using the neighborhood sample points, wherein the neighborhood sample point set is a set of reference points for pixel similarity judgment, and the first preset threshold is used to indicate the number of pixels in the corresponding neighborhood sample point set whose pixel value changes within the pixel value error range when determining that any pixel is similar to a pixel in the current sample set.

[0040] In one embodiment, after training a preset background model according to preset motion detection requirements and the sample set to obtain a background model, the method further includes: obtaining a target background point set from the background points identified by the background model according to a time sampling factor; adding the target background point set to the sample set to update the background points in the sample set; and using the updated sample set to verify the background model.

[0041] In other words, to improve the detection accuracy of the background model under constantly changing conditions, a corresponding update strategy can be set to update the sample set. Specifically, this is achieved by determining the background point detected. The probability is used to update the sample value of the pixel, and the neighboring pixels of the pixel are also used. The probability is used to update the sample value, where l represents the time sampling factor. Optionally, the background model can set its value to 16. Optionally, for a detected foreground point, if it is identified as a foreground point for 20 consecutive frames, it is set as a background point and processed accordingly. The algorithm updates the sample value with a probability of [a certain value]. It should be noted that this update strategy utilizes the spatial propagation characteristics of pixel values, which can effectively avoid the generation of ghosting regions. Furthermore, since it updates only one pixel value randomly, the update of the sample value during the algorithm process is independent of time, which greatly improves the detection accuracy.

[0042] To better understand the technical solutions of the embodiments and optional embodiments of the present invention, the flow of the above-described target area detection method is explained below with reference to examples, but it is not intended to limit the technical solutions of the embodiments of the present invention.

[0043] As an optional implementation, a deep neural network for infrared-visible light fusion and its modeling method are provided. This method extracts features using both infrared and visible light encoders, fuses the encoder features, and outputs the fused results to a decoder module for dual-light fusion recognition. This avoids the shortcomings of using a single infrared or visible light camera to locate hazardous sources in power monitoring. However, the constructed network model can only function as a classification network to determine whether a hazardous target is present based on the segmented infrared-visible light image dataset. The loss function only uses cross-entropy loss and does not define a bounding box regression component, therefore it cannot track and locate specific targets and is not suitable for target location in all-day intelligent monitoring scenarios.

[0044] As an optional implementation, an infrared-visible light fusion recognition method based on a symmetric fusion network is also provided. First, deep feature extraction is performed on existing infrared and visible light images to obtain infrared and visible light image feature maps. Then, tensor concatenation and feature fusion are performed on the two feature maps respectively to obtain infrared fused and visible light fused images. Finally, the feature ratio of the two fused images is adjusted by a weight offset coefficient to obtain the final ship detection and recognition result. Although adaptive weight allocation is added to adapt to daytime visible light scenes and low-brightness nighttime scenes, the fused features are not further extracted and utilized. Furthermore, many missed detections and false detections occur in rainy, snowy, and occluded scenes.

[0045] An optional embodiment of this invention proposes a target detection model based on infrared and visible light image fusion. This model deeply extracts and fuses features from infrared and visible light images, while simultaneously leveraging a motion detection module to improve the accuracy of target detection in complex scenes. It enables target detection, localization, and classification in both daytime and nighttime conditions, as well as in complex environments. The target detection model fuses visible light and infrared images at the feature level to obtain the recognition results from the target detection module. After obtaining the infrared image detection results from the motion detection module, a second fusion is performed with the target detection results to obtain the final target detection result. Therefore, it can improve the device's fault tolerance and recognition accuracy, achieving intelligent target detection around the clock in various complex scenes, both day and night.

[0046] As an optional implementation method, Figure 3This is a schematic diagram of the infrared-visible light fusion deep neural network corresponding to the target detection model in an optional embodiment of the present invention. The infrared-visible light fusion deep neural network processes and fuses the input infrared image and the visible light image to achieve deep feature extraction and reuse of the infrared image and the visible light image. Optionally, the infrared-visible light fusion deep neural network is a DLA34 network structure. The main body of this network consists of three modules: feature extraction and fusion module 32, HDA (hierarchical deep aggregation) UP feature fusion module 34, and IDA (iterative deep aggregation) UP feature fusion module 36.

[0047] Figure 4 This is a flowchart illustrating an optional embodiment of the target detection model of the present invention. The specific processing flow is as follows:

[0048] Step S302: Acquire visible light and infrared images of the current scene using visible light and infrared cameras, and perform registration and calibration. Use the calibrated infrared and visible light images as two inputs to the infrared-visible light fusion deep neural network to achieve deep feature extraction and reuse of the infrared and visible light images.

[0049] Optionally, the registration and calibration judgment is as follows: the edges are used as the correlation features for the registration of the two types of images. First, the grayscale information of the edges is preprocessed and enhanced. The edges are extracted using the Canny operator and the affine transformation parameters are calculated using boundary correlation to achieve the registration and calibration of the two images.

[0050] Step S304: The feature extraction and fusion module 32 performs feature extraction on the input calibrated infrared image and the calibrated visible light image respectively, and after the extraction is completed, the extracted infrared feature map and visible light feature map are fused by tensor splicing as the input of the subsequent feature extraction and fusion module.

[0051] Optionally, the feature extraction and fusion module 32 includes three convolutional layer modules with a size of 7*7*16 and one tensor concatenation module.

[0052] Step S306: The HDAUP feature fusion module 34 takes the infrared-visible light fused feature map output by the feature extraction and fusion module 32 as input, and performs downsampling extraction of features in the infrared-visible light fused feature map in four stages. HDAUP emphasizes feature extraction and fusion of blocks within the four stages, using the idea of ​​skip-connection, so that the current block can aggregate feature maps from previous layers and restore the size through the deconvolution module. The IDAUP feature fusion module 36, based on the features restored to size by the HDAUP feature fusion module 34 through the deconvolution module, uses aggregation nodes to aggregate features from the bottom layer to the top layer. The aggregation formula for IDA is... I represents the entire IDAUP feature fusion module, N represents the number of aggregation nodes, and N(x1,x2) represents an aggregation node with inputs x1 and x2. It fuses features from different stages through deconvolution.

[0053] Step S308: The head part of the output of the infrared-visible light fusion deep neural network consists of three parts: heatmap prediction category hm_head, width and height prediction wh_head, and pixel restoration offset prediction reg_head. Similarly, the loss function also consists of these three parts, with a weight ratio of hm_head:reg_head:wh_head = 1:1:0.1. The loss function of hm_head predicts the center point of the bounding box by dividing the heatmap samples into easy and difficult samples, assigning a larger loss function to the difficult samples and a smaller loss function to the easy samples. The loss function of reg_head is the pixel value of the downsampled original image after rounding to account for the deviation. wh_head is the loss calculated using the L1 loss function to predict the width and height of the bounding box.

[0054] It should be noted that, through the HDAUP feature fusion module and the IDAUP feature fusion module, the infrared and visible light fused feature map undergoes feature re-extraction and feature fusion between and within each stage, which improves the accuracy of recognition and enhances robustness.

[0055] Furthermore, using the aforementioned DLA34 network model can better ensure the model's accuracy, reducing the number of missed and false detections. Moreover, due to the strong anti-interference capabilities of infrared images, targets can still be detected in extreme weather conditions such as rain, snow, or low visibility at night, and can be used as input to the motion detection module. Fusing the recognition results from the target detection network and the motion detection module can further improve recognition accuracy.

[0056] Step S310: The above target detection model further includes a motion detection module, used to perform motion detection on the infrared image. The motion detection module uses the ViBe (visual background extractor) algorithm to extract the background. It establishes a sample set to store all pixels, saving the past pixel values ​​of each pixel and the pixel values ​​of its neighbors. It determines whether a pixel belongs to the background by comparing the current pixel value with the historical values ​​of the sample.

[0057] Optionally, the background model part of the above object detection model is designed as follows: V(x,y) represents the current corresponding pixel value, M(x,y)={V1(x,y),...V n (x,y) represents the historical sample set, and R is the range of values ​​for the difference between the current pixel value and the historical sample value, which is set to 20 in this background model. First, the number of neighboring sample points n in the sample set is selected as 50, which increases adaptability to complex scenes while ensuring sufficient memory usage. The threshold th is set to 43. The pixel value V(x,y) is subtracted from the points in the historical sample set M(x,y), and ±R is the allowed fluctuation range. If the number of points N within this range is greater than th, it means that the pixel is similar to a historical pixel and can be classified as a background point.

[0058] Optionally, the background modeling initialization method is to select the first frame of the video frame as the background model and select 50 points around each pixel as the sample set, which reduces the amount of computation while storing the spatiotemporal information of the video.

[0059] Optionally, the determination of foreground points is similar to that of background points. Based on the background point determination process described above, the threshold T = n - th for determining foreground points can be set to 7. That is, if the number N1 obtained by subtracting the pixel value V(x,y) from the points in the historical sample set M(x,y) that is greater than R or less than -R is greater than T, then the pixel is determined to be a foreground point.

[0060] Optionally, the ViBe algorithm described above can also be configured with a corresponding update strategy to update the sample set. Specifically, this is achieved by determining the status of each detected background point. The probability is used to update the sample value of the pixel, and the neighboring pixels of the pixel are also used. The probability is used to update the sample value, where l represents the time sampling factor, which is set to 16 in this object detection model. Simultaneously, for detected foreground points, if they are identified as foreground points for 20 consecutive frames, they are set as background points and treated similarly. The algorithm updates the sample value with a probability of [a certain value]. It should be noted that this update strategy utilizes the spatial propagation characteristics of pixel values, effectively avoiding the generation of ghost regions. Furthermore, since it updates only one pixel value randomly, the update of the sample value during the algorithm process is independent of time, greatly improving the accuracy of detection.

[0061] Step S312: Determine the motion detection result of the motion detection module on the infrared image;

[0062] Step S314: Fuse the output of the infrared-visible light fusion deep neural network with the motion detection results of the infrared image;

[0063] Step S316: Output the final detection result of the target detection model.

[0064] The above embodiments provide a target detection model. By adding a motion detection module to the original infrared-visible light fusion recognition, the strong anti-interference capability of infrared images enhances the target detection accuracy of the device in complex environments. Furthermore, by fusing the results of the infrared-visible light fusion recognition network with the results of infrared image motion detection recognition, the false detection rate and false negative rate in complex scenes are reduced, enabling more effective all-day monitoring in various conditions such as day and night, rain, snow, fog, and obstructed scenes. In practice, after using a simple convolutional network module to extract and fuse features from visible light and infrared images, a DLA34 network is introduced to further extract and utilize the fused features. Simultaneously, feature map fusion is performed between layers and stages within the network, resulting in deeper extraction and utilization of infrared and visible light image features. This enhances the robustness and accuracy of the detection model, enabling target localization, detection, and classification in various complex scenes. Compared to existing technologies, it offers higher accuracy and recall, broadening its application scenarios.

[0065] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0066] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0067] According to another aspect of the present invention, a target region detection apparatus for implementing the above-described target region detection method is also provided. For example... Figure 5 As shown, the device includes:

[0068] The receiving module 52 is used to receive visible light images and infrared images sent by the target device, wherein the visible light images and the infrared images are obtained from the same video frame, and the same video frame is any frame image in the video frame obtained by the target device to capture images of the target area.

[0069] Target detection module 54 is used to input the visible light image and the infrared image into the target detection network to obtain a first detection result corresponding to the target region;

[0070] The fusion module 56 is used to fuse the first detection result with the second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target area.

[0071] Through the above steps, a target detection network is used to analyze and fuse visible light and infrared images to obtain the first detection result of the target area. This first detection result is then fused with the second detection result obtained by motion detection of the infrared image using a background model to obtain the third detection result. This process allows for deeper extraction and utilization of features from both infrared and visible light images, enhancing the robustness and accuracy of the detection model. Consequently, it achieves the goal of accurately locating, detecting, and classifying target objects within the target area, thereby improving the technical efficiency of target area detection. This enables the target device to more effectively handle all-day monitoring in various conditions, including day and night, rain, snow, fog, and obstructed scenes. This solves the technical problem of low accuracy in target area detection in complex scenarios in existing technologies.

[0072] Optionally, the above-mentioned target detection module is further configured to input the visible light image and the infrared image into the feature extraction and fusion module in the target detection network to obtain a fused feature map of the visible light image and the infrared image output by the feature extraction and fusion module, wherein the feature extraction and fusion module includes multiple cascaded convolutional layer modules and a tensor concatenation module; input the fused feature map into the hierarchical depth aggregation module in the target detection network to obtain multiple upsampled feature maps output by the hierarchical depth aggregation module; input the multiple upsampled feature maps into the iterative depth aggregation module in the target detection network to obtain predicted bounding box information output by the iterative depth aggregation module, and use the predicted bounding box information as the first detection result of the target detection network for the target region.

[0073] Optionally, the target detection module further includes: a feature extraction unit, used to extract features from the visible light image through the plurality of convolutional layer modules to obtain a visible light feature map corresponding to the visible light image; to extract features from the infrared image through the plurality of convolutional layer modules to obtain an infrared feature map corresponding to the visible light image; and to use the tensor stitching module to stitch the visible light feature map and the infrared feature map together to determine a fused feature map of the visible light image and the infrared image.

[0074] Optionally, the above-mentioned target detection module further includes: a first feature fusion unit, used to downsample and extract the fused feature map through the first-layer downsampling submodule of the hierarchical depth aggregation module to obtain a first-stage feature map corresponding to the fused feature map; wherein, the downsampling extraction is used to instruct the fused feature map to be divided into multiple basic blocks according to the preset block size of the first-layer downsampling submodule, and to perform feature fusion on each of the multiple basic blocks; the first-stage feature map is input into the second-layer downsampling submodule of the hierarchical depth aggregation module for downsampling extraction to obtain a second-stage feature map corresponding to the fused feature map, and the second-stage feature map is partially aggregated through a preset residual convolutional layer to determine the second target-level feature map that is then input into the third-layer downsampling submodule of the hierarchical depth aggregation module for downsampling extraction; when all the downsampling submodules in the hierarchical depth aggregation module have completed downsampling extraction, each target-level feature map processed by the residual convolutional layer is upsampled to determine multiple upsampled feature maps output by the hierarchical depth aggregation module.

[0075] In short, the hierarchical deep aggregation module divides the infrared-visible light fusion feature map into multiple levels for downsampling and extraction. Within each level, it performs block feature extraction and fusion, and uses the idea of ​​skip connections. By passing the feature map of the previous level through a max pooling layer to skip to the downsampling extraction of the next level for feature fusion, the feature map of the previous level is aggregated into the current block and its size is restored by the deconvolution module. Then, according to the number of levels of the hierarchical deep aggregation module, it can output an upsampled feature map after processing the infrared-visible light fusion feature map by a number corresponding to the number of levels.

[0076] Optionally, the above-mentioned target detection module further includes: a second feature fusion unit, used to perform feature fusion on multiple upsampled feature maps output by the hierarchical deep aggregation module through the accumulating convolution module in the iterative deep aggregation module; and to determine the predicted bounding box information based on the result of the feature fusion, wherein the predicted bounding box information includes at least one of the following: the heatmap prediction category of the predicted bounding box, the length and width prediction of the predicted bounding box, and the pixel restoration offset prediction of the predicted bounding box.

[0077] Optionally, the above device further includes: a background detection module, used to take the infrared image of the first frame of the video frame acquired by the target device as a standard infrared image, and determine the background model of the target area based on the standard infrared image; and to judge each pixel in the infrared images of the other frames of the video frame except the first frame through the background model, so as to segment the foreground and background pixels in the infrared images of the other frames.

[0078] Optionally, the background detection module described above is further configured to determine the difference between the current pixel value of each pixel in the infrared image of the other frames and the pixel value in the sample set corresponding to the background model; if the difference is within the pixel value error range corresponding to the background model, the current pixel is determined to be a target background point; if the difference is outside the pixel value error range corresponding to the background model, the current pixel is determined to be a target foreground point.

[0079] Optionally, the background model is designed as follows: V(x,y) represents the current pixel value, M(x,y)={V1(x,y),...V n(x,y) represents the historical sample set, and R is the range of the difference between the current pixel value and the historical sample value, which is set to 20 in this background model. First, the number of neighborhood sample points n in the sample set is selected as 50, which increases adaptability to complex scenes while ensuring sufficient memory usage. The threshold th is set to 43. The pixel value V(x,y) is subtracted from the points in the historical sample set M(x,y), and ±R is the allowed fluctuation range. If the number of points N>th within this range is similar to historical pixels, it is considered a background point. The judgment of foreground points is similar to that of background points. Based on the background point judgment process described above, the threshold T = n-th for judging foreground points can be set to 7. That is, if the number N1 obtained by subtracting the pixel value V(x,y) from the points in the historical sample set M(x,y) that is greater than R or less than -R is greater than T, then the pixel is judged as a foreground point.

[0080] Optionally, the background detection module described above is further configured to determine the pixel values ​​corresponding to all pixels in the standard infrared image and / or the pixel values ​​of the neighboring pixels of the pixels, thereby obtaining a sample set of the standard infrared image; and to train a preset background model according to preset motion detection requirements and the sample set, thereby obtaining a background model.

[0081] Optionally, the aforementioned background detection module is further configured to determine the allowed pixel value error range of the preset background model based on the pixel difference and the preset error threshold of the motion detection requirement, wherein the pixel difference is the difference between the maximum and minimum pixel values ​​of the sample set; determine a target number of neighborhood sample point sets from the sample set according to the preset motion detection requirement, and obtain a first preset threshold for determining any pixel as a background point using the neighborhood sample points, wherein the neighborhood sample point set is a set of reference points for pixel similarity judgment, and the first preset threshold is used to indicate the number of pixels in the corresponding neighborhood sample point set whose pixel value changes within the pixel value error range when determining that any pixel is similar to a pixel in the current sample set.

[0082] Optionally, the background detection module further includes an update unit, which is used to obtain a set of target background points from the background points identified by the background model according to the time sampling factor; add the set of target background points to the sample set to update the background points in the sample set; and use the updated sample set to verify the background model.

[0083] In other words, to improve the detection accuracy of the background model under constantly changing conditions, a corresponding update strategy can be set to update the sample set. Specifically, this is achieved by determining the background point detected. The probability is used to update the sample value of the pixel, and the neighboring pixels of the pixel are also used. The probability is used to update the sample value, where l represents the time sampling factor. Optionally, the background model can set its value to 16. Optionally, for a detected foreground point, if it is identified as a foreground point for 20 consecutive frames, it is set as a background point and processed accordingly. The algorithm updates the sample value with a probability of [a certain value]. It should be noted that this update strategy utilizes the spatial propagation characteristics of pixel values, which can effectively avoid the generation of ghosting regions. Furthermore, since it updates only one pixel value randomly, the update of the sample value during the algorithm process is independent of time, which greatly improves the detection accuracy.

[0084] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0085] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0086] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0087] S1. Receive visible light image and infrared image sent by target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in the video frame obtained by the target device to capture images of the target area;

[0088] S2. Input the visible light image and the infrared image into the target detection network to obtain the first detection result corresponding to the target region;

[0089] S3. The first detection result is fused with the second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target area.

[0090] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0091] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0092] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0093] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0094] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0095] S1. Receive visible light image and infrared image sent by target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in the video frame obtained by the target device to capture images of the target area;

[0096] S2. Input the visible light image and the infrared image into the target detection network to obtain the first detection result corresponding to the target region;

[0097] S3. The first detection result is fused with the second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection of the infrared image of the target area.

[0098] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0099] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0100] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0101] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method of detecting a target region, characterized by, The method comprises: receiving a visible light image and an infrared image sent by a target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in a video frame obtained by the target device for image acquisition of a target region; inputting the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region; fusing the first detection result and a second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by motion detection on the infrared image of the target region; wherein, before fusing the first detection result and the second detection result corresponding to the infrared image to obtain a third detection result, the method further comprises: taking the infrared image of the first frame in the video frame obtained by the target device as a standard infrared image; determining the pixel value corresponding to all pixel points and / or the pixel value of the neighbor points corresponding to the pixel points in the standard infrared image to obtain a sample set of the standard infrared image; training a preset background model according to the preset motion detection requirement and the sample set to obtain a background model, wherein the background model is used for identifying foreground points and background points in the infrared image for motion detection.

2. The method of claim 1, wherein, inputting the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region, comprising: inputting the visible light image and the infrared image into a feature extraction and fusion module in the target detection network to obtain a fusion feature map of the visible light image and the infrared image output by the feature extraction and fusion module, wherein the feature extraction and fusion module comprises a plurality of convolution layer modules and a tensor splicing module connected in cascade; inputting the fusion feature map into a hierarchical depth aggregation module in the target detection network to obtain a plurality of up-sampling feature maps output by the hierarchical depth aggregation module; inputting the plurality of up-sampling feature maps into an iterative depth aggregation module in the target detection network to obtain prediction bounding box information output by the iterative depth aggregation module, and taking the prediction bounding box information as the first detection result of the target region by the target detection network.

3. The method of claim 2, wherein, inputting the visible light image and the infrared image into a feature extraction and fusion module in the target detection network to obtain a fusion feature map of the visible light image and the infrared image output by the feature extraction and fusion module, comprising: extracting features of the visible light image through the plurality of convolution layer modules to obtain a visible light feature map corresponding to the visible light image; extracting features of the infrared image through the plurality of convolution layer modules to obtain an infrared feature map corresponding to the visible light image; splicing features of the visible light feature map and the infrared feature map using the tensor splicing module to determine the fusion feature map of the visible light image and the infrared image.

4. The method of claim 2, wherein, inputting the fusion feature map into a hierarchical depth aggregation module in the target detection network to obtain a plurality of up-sampling feature maps output by the hierarchical depth aggregation module, comprising: The fusion feature map is down-sampled and extracted by a first layer down-sampling module of the hierarchical deep aggregation module to obtain a first stage feature map corresponding to the fusion feature map; wherein the down-sampling extraction is used to indicate that the fusion feature map is divided into a plurality of basic blocks according to a preset block size of the first layer down-sampling module, and feature fusion is performed on each of the plurality of basic blocks; The first stage feature map is input into a second layer down-sampling module of the hierarchical deep aggregation module for down-sampling extraction to obtain a second stage feature map corresponding to the fusion feature map, and the second stage feature map is aggregated by a preset residual convolution layer to determine a second target level feature map input into a third layer down-sampling module of the hierarchical deep aggregation module for down-sampling extraction; In the case that all the down-sampling modules of all levels in the hierarchical deep aggregation module complete down-sampling extraction, each target level feature map processed by the residual convolution layer is up-sampled to determine a plurality of up-sampled feature maps output by the hierarchical deep aggregation module.

5. The method of claim 2, wherein, The plurality of up-sampled feature maps are input into an iterative deep aggregation module in the target detection network to obtain prediction bounding box information output by the iterative deep aggregation module, including: The plurality of up-sampled feature maps output by the hierarchical deep aggregation module are fused by an accumulation convolution module in the iterative deep aggregation module; According to the result of the feature fusion, prediction bounding box information is determined, wherein the prediction bounding box information includes at least one of the following: a heat map prediction class of a prediction bounding box, a length-width prediction of a prediction bounding box, and a pixel restoration offset prediction in a prediction bounding box.

6. The method of claim 1, wherein, After training the preset background model according to the preset motion detection requirement and the sample set, the method further comprises: Judging each pixel point in the infrared image of the other frames in the video frame except the first frame by the background model to segment out the foreground points and the background points in the infrared image of the other frames.

7. The method of claim 6, wherein, Judging each pixel point in the infrared image of the other frames in the video frame except the first frame by the background model to segment out the foreground points and the background points in the infrared image of the other frames, including: Determining the difference between the current pixel value of each pixel point in the infrared image of the other frames and the pixel value in the sample set corresponding to the background model; In the case that the difference is within the pixel value error interval corresponding to the background model, determining that the current pixel point is a target background point; In the case that the difference is outside the pixel value error interval corresponding to the background model, determining that the current pixel point is a target foreground point.

8. The method of claim 1, wherein, Training the preset background model according to the preset motion detection requirement and the sample set, including: Determining the pixel value error interval allowed by the preset background model according to the pixel difference and the error threshold of the preset motion detection requirement, wherein the pixel difference is the difference between the maximum pixel value and the minimum pixel value of the sample set; According to the preset motion detection requirement, a target number of neighborhood sample point sets are determined from the sample set, and a first preset threshold value for determining any pixel point as a background point using the neighborhood sample points is obtained, wherein the neighborhood sample point set is a set of reference points for pixel point similarity judgment, and the first preset threshold value is used to indicate the number of pixel points in the corresponding neighborhood sample point set whose pixel value changes within the pixel value error interval when determining any pixel point and a pixel point in the current sample set as similar pixel points.

9. The method of claim 1, wherein, According to the preset motion detection requirement and the sample set, a preset background model is trained to obtain a background model, and the method further comprises: According to a time sampling factor, a target background point set is obtained from the background points identified from the background model; The target background point set is added to the sample set to update the background points in the sample set, and the updated sample set is used to verify the background model.

10. A device for detecting a target region, characterized by Comprise: The receiving module is configured to receive a visible light image and an infrared image sent by a target device, wherein the visible light image and the infrared image are obtained from the same video frame, and the same video frame is any frame image in a video frame obtained by the target device for image acquisition of a target region. The target detection module is configured to input the visible light image and the infrared image into a target detection network to obtain a first detection result corresponding to the target region. The fusion module is configured to fuse the first detection result and a second detection result corresponding to the infrared image to obtain a third detection result, wherein the second detection result is determined by performing motion detection on the infrared image of the target region. The device further comprises a background detection module configured to, before fusing the first detection result and the second detection result corresponding to the infrared image to obtain a third detection result, take the infrared image of the first frame in the video frame obtained by the target device as a standard infrared image, determine the pixel values corresponding to all pixel points and / or the pixel values of the corresponding neighbor points of the pixel points in the standard infrared image to obtain a sample set of the standard infrared image, and train a preset background model according to a preset motion detection requirement and the sample set to obtain a background model, wherein the background model is used to identify foreground points and background points in the infrared image for motion detection.

11. A computer readable storage medium, characterized in that, The computer-readable storage medium comprises a stored program, wherein the program executes the method described in any one of claims 1 to 9 when running.

12. An electronic device comprising a memory and a processor, characterized in that The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 9 by using the computer program.

Images