Object detection method, device and system for billion-pixel fusion video

Abstract

The present application relates to a method, apparatus, billion-pixel computational imaging system, computer equipment, storage medium, and computer program product for target detection in billion-pixel fusion videos. The method comprises: acquiring multiple local videos captured by an array camera; reducing the size of the local video image of the current frame, inputting it into a first target detection model for target detection, and obtaining a first detection result; performing motion target detection on the local video using a frame difference method to determine a target detection area that meets a small target size condition; inputting the image of the target detection area into a second target detection model for target detection, and obtaining a second detection result; and obtaining a target detection result for the fused video image of the current frame of the fused video based on the first detection result and the second detection result. The present method can improve the target detection accuracy and efficiency of billion-pixel fusion videos.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting a target in a billion-pixel fusion video, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its high reliability, timely nature, and ease of viewing. When monitoring large areas, billion-pixel computational imaging systems based on array cameras have emerged to achieve ultra-high-definition video surveillance of large scenes. These systems use multiple long-focus, narrow-field-of-view (FOV) sub-cameras within the array to capture multiple HD partial videos of the monitored scene. These partial videos are then stitched together and fused to produce ultra-high-definition fused video with a large or wide field of view, reaching up to billion-pixel resolution. These systems are suitable for video capture and security monitoring of large scenes such as airports, highways, parks, sports stadiums, border crossings, and ocean surfaces.

[0003] In related technologies, billion-pixel computational imaging systems can also provide intelligent target detection capabilities, using target detection models to detect surveillance targets in fused video images in real time, such as people, cars, airplanes, birds, drones, etc. Due to the large amount of data in billion-pixel fused videos, if target detection is performed directly on billion-pixel fused video images or all local video images, the detection efficiency is low, and the hardware computing power requirements and computing resource consumption are high. Therefore, in actual applications, in order to improve target detection efficiency and save computing resources, the size of each local video image is usually reduced, such as downsampling (downsampling, reducing) an image with a resolution of 3840*2160 to a resolution of 640*360, and then inputting it into the target detection model for detection.

[0004] However, the method of reducing the size of local video images before performing target detection will significantly reduce the detection accuracy, especially for small targets with a small pixel ratio such as drones and flying birds, the missed detection rate or false detection rate is high. Therefore, this method is difficult to achieve both high detection accuracy and high detection efficiency. Summary of the Invention

[0005] Based on this, it is necessary to provide a target detection method, device, computer equipment, computer-readable storage medium and computer program product that can improve the target detection accuracy and detection efficiency of billion-pixel fusion videos in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for object detection in a billion-pixel fusion video. The method comprises:

[0007] Obtain multiple local videos captured by array cameras;

[0008] Resizing the local video image of the current frame of each of the local videos, and inputting the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result;

[0009] Performing moving target detection on the local video using a frame difference method to obtain candidate detection regions where each moving target is located in the local video image of the current frame, and determining a target detection region that meets a small target size condition from each candidate detection region;

[0010] Inputting the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result;

[0011] Based on the first detection result and the second detection result, a target detection result of a fused video image of a current frame of the fused video is obtained; the fused video is obtained by splicing and fusing the multiple local videos.

[0012] In one embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame;

[0013] The obtaining, based on the first detection result and the second detection result, a target detection result of the fused video image of the current frame of the fused video, includes:

[0014] Performing coordinate conversion on the local position information of the first target object and the second target object according to the coordinate mapping relationship between each of the partial videos and the fused video to obtain fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame;

[0015] Searching, based on the fusion position information of the first target object and the second target object, for a first target object and a second target object that meet a fusion position intersection condition, and determining whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, merging the fusion position information of the first target object and the second target object that meet the fusion position intersection condition as the fusion position information of the target merging object;

[0016] The first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

[0017] In one embodiment, obtaining the target detection result of the fused video image of the current frame of the fused video based on the first detection result and the second detection result includes:

[0018] reducing the size of the fused video image of the current frame of the fused video and inputting the image into a third target detection model for target detection, obtaining a third detection result; the third detection result includes the type of the detected third target object and fused position information of the third target object in the coordinate system of the fused video image of the current frame;

[0019] An object detection result of the current frame fused video image is obtained based on the first detection result, the second detection result, and the third detection result.

[0020] In one embodiment, the step of performing moving target detection on the local video using a frame difference method to obtain candidate detection areas where each moving target is located in the local video image of the current frame includes:

[0021] For each of the partial videos, obtaining a binary differential image corresponding to the current frame partial video image based on the current frame partial video image and the historical frame partial video images of the partial video;

[0022] Connected domain analysis is performed on the differential image to determine connected areas formed by target pixel points, and candidate detection areas where each moving target is located are determined based on each connected area; the target pixel points are pixel points corresponding to the moving target.

[0023] In one embodiment, obtaining a binary difference image corresponding to the local video image of the current frame based on the local video image of the current frame and the local video image of the historical frame includes:

[0024] Determine a historical frame local video image that is a preset number of frames away from the current frame local video image in the partial video as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image;

[0025] Performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial differential images, and performing binarization processing on each of the initial differential images to obtain a plurality of binarized initial differential images;

[0026] The binarized initial differential images are superimposed, and the superimposed images are morphologically processed to obtain a binarized differential image corresponding to the local video image of the current frame.

[0027] In one embodiment, performing connected domain analysis on the differential image to determine a connected region formed by target pixels includes:

[0028] Using the initial serial number of each target pixel in the differential image as the current serial number for the first update process, the initial serial number being an ascending or descending serial number pre-assigned to each target pixel in the differential image;

[0029] For each target pixel, searching for other target pixels connected in the extension direction of adjacent pixels of each target pixel in the differential image as first associated pixels, and using the minimum or maximum current sequence number of the target pixel and each of the first associated pixels as the target sequence number of the target pixel;

[0030] Determine whether the target sequence number and the initial sequence number of the target pixel point are consistent; if the target sequence number and the initial sequence number of the target pixel point are consistent, determine the target sequence number as the updated sequence number of the target pixel point;

[0031] If the target sequence number and the initial sequence number of the target pixel point are inconsistent, another target pixel point whose initial sequence number is consistent with the target sequence number of the target pixel point is used as a second associated pixel point, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point;

[0032] The current serial number of each target pixel point is updated to the update serial number, and the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image is returned to execute for each target pixel point until the preset number of update processing times is reached, and the update serial number determined after the last update processing is used as the matching serial number, and the area composed of the target pixel points with the same matching serial number is determined as a connected area.

[0033] In one embodiment, determining the update sequence number of the target pixel point according to the target sequence number and the initial sequence number of the second associated pixel point includes:

[0034] Compare whether the target serial number and the initial serial number of the second associated pixel point are the same. If they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, use other target pixel points whose initial serial number is consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

[0035] In a second aspect, the present application also provides an object detection device for billion-pixel fusion video. The device comprises:

[0036] An acquisition module is used to acquire multiple local videos captured by array cameras;

[0037] A first detection module is configured to reduce the size of a local video image of a current frame of each of the local videos, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result;

[0038] a second detection module configured to perform moving target detection on the local video using a frame difference method, obtain candidate detection regions where each moving target is located in the local video image of the current frame, and determine a target detection region that meets a small target size condition from each candidate detection region; input the image of the target detection region in the local video image of the current frame into a second target detection model for target detection, and obtain a second detection result;

[0039] The third detection module is used to obtain the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusion of the multiple local videos.

[0040] In a third aspect, the present application also provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera, a server, and a display terminal. The array camera includes multiple sub-cameras, wherein:

[0041] The array camera is used to capture a target scene through multiple sub-cameras and send the captured multiple local videos to the server;

[0042] The server is used to splice and fuse the acquired partial videos to obtain a fused video;

[0043] The server is further configured to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; perform motion target detection on the local video using a frame difference method to obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine a target detection area that meets a small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result; and obtain a target detection result of the fused video image of the current frame of the fused video based on the first detection result and the second detection result;

[0044] The server is further configured to send the fused video and the target detection result to the display terminal.

[0045] In a fourth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0046] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0047] In a sixth aspect, the present application further provides a computer program product, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0048] The above-mentioned target detection method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for billion-pixel fusion video achieve rapid detection of larger-sized targets by reducing the local video image and inputting it into the first target detection model for target detection, and perform motion target detection on the local video image to determine the candidate detection area where the moving target is located, and input the image of the target detection area that meets the small target size condition into the second target detection model to achieve accurate detection of smaller targets. Moreover, since the image size of the target detection area is small, it serves as the input of the second target detection model, and the model needs to process a small amount of data, so the detection efficiency is high. The two types of detection results are then merged to obtain the target detection result of the billion-pixel fusion video. Thus, the technical effect of high accuracy and high detection efficiency for target detection of billion-pixel fusion video can be achieved.

[0049] In addition, since small targets account for a small proportion of pixels in billion-pixel fusion videos and local video images, they contain too little color information such as RGB and discriminative features. Even if small target detection is performed directly on uncompressed billion-pixel fusion videos or local video images, the false detection rate is still high. This method uses motion target detection and detection area size condition screening to preliminarily determine the target detection area where small targets may be located, and then uses the image of this target detection area for small target detection. Since small targets account for a large proportion of pixels in this target detection area, that is, the image of the target detection area mainly contains the characteristic information of small targets, compared to detecting small targets in a whole large image, this method can further improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of an example of a billion-pixel computational imaging system;

[0051] Figure 2 1. A schematic flow chart of a method for object detection in a billion-pixel fusion video according to an embodiment;

[0052] Figure 3 A schematic diagram of a process for determining a target detection result in one embodiment;

[0053] Figure 4 A schematic diagram of a process for determining a target detection result in another embodiment;

[0054] Figure 5 A schematic diagram of a process for determining a candidate detection area in one embodiment;

[0055] Figure 6 This is a structural block diagram of an object detection device for billion-pixel fusion video in one embodiment;

[0056] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] The object detection method of the billion-pixel fusion video provided in the embodiment of the present application can be applied to Figure 1The illustrated exascale pixel computational imaging system includes an array camera 102 and a display terminal 106, each communicating with a server 104 via a network. Display terminal 106 can be, but is not limited to, various electronic devices with display units, such as personal computers, laptops, smartphones, tablets, smart displays, IoT devices, and portable wearable devices. Server 104 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0059] In one embodiment, Figure 2 As shown in the figure, a target detection method for billion-pixel fusion video is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0060] Step 201: Acquire multiple channels of local videos captured by array cameras.

[0061] The array camera 102 may be an array camera in the prior art or other new array cameras. The present application does not limit the specific structure of the array camera. The array camera includes multiple sub-cameras with different shooting angles, such as M (rows) * N (columns), which respectively shoot local high-definition videos (such as 4K videos) of the target scene. The shooting ranges of adjacent sub-cameras usually have overlapping areas. In some examples, the array camera may also include a panoramic camera. Compared with other sub-cameras that shoot local videos, the panoramic camera is a camera with a short focal length and a wide field of view. Its field of view can cover the field of view of other sub-cameras and is used to assist in establishing image stitching models and color correction. The array camera can send the captured local videos to the server through a communication module.

[0062] After receiving the partial videos from each channel, the server can use a pre-deployed image stitching and fusion model to stitch, fuse, color correct, and crop a set of partial video images captured by each sub-camera at the same or similar times. This produces an ultra-high-definition fused video image with a large or wide field of view and up to 100 million pixels. The fused video images can then be generated into a fused video stream and sent to a display terminal for display.

[0063] It is understandable that the partial video can be a real-time video or a pre-recorded video, and this application does not limit this.

[0064] In step 202 , the local video image of the current frame of each local video is reduced in size, and the reduced-size local video image of the current frame is input into a first target detection model for target detection to obtain a first detection result.

[0065] After receiving the partial video, the server can perform target detection frame by frame, or perform target detection at a preset time period (e.g., 1 second) or at a preset number of frame intervals. The server can downsample the currently processed partial video image (the current frame partial video image) to reduce the size of the current frame partial video image to the target size.

[0066] Then, the server can input the current frame local video image (reduced image) of the target size into the first target detection model for target detection. The first target detection model can be established using target detection algorithms such as Yolo and Fast-RCNN, and trained using sample images of the target size. The type of target object detected by the first target detection model can be one or more of people, cars, airplanes, flying birds, drones, flames, thick smoke, etc., which can be set as needed. The output result of the first target detection model can be the type and position information of each target object in the reduced local video image (such as the coordinates (x, y, w, h) of the rectangular detection frame), and then the position information can be converted into local position information in the coordinate system of the uncompressed local video image, and the type and local position information of the target object are used as the first detection result.

[0067] It should be noted that the shooting time of the current frame local video image of each local video is the same or similar (there may be a certain exposure time difference). The current frame fused video image of the fused video is obtained by splicing and fusing the current frame local video images of each local video.

[0068] Step 203 , using the frame difference method to perform motion target detection on the local video, obtain candidate detection areas where each motion target is located in the local video image of the current frame, and determine the target detection area that meets the small target size condition from each candidate detection area.

[0069] Among them, the small target size condition can be set according to the number of pixels or pixel ratio of the small target in the local video image. The small target is a target with a smaller pixel ratio in the image relative to other target objects with a larger pixel ratio in the local video image. It can be a target object with a smaller absolute size (such as a flying bird, a drone, etc.), or it can be other target objects with a longer shooting distance, such as a car or a person in the distance, or a large airplane in the sky farther away, so the small target has nothing to do with the type of target object. The number of pixels less than the first threshold, or the number of pixels after the size reduction processing in step 202 is less than the second threshold, can be used as a small target size condition. The specific threshold can be pre-set in combination with factors such as the distance of the shooting scene, the absolute size of the target object to be monitored, and the reduction ratio of the size reduction processing.

[0070] The server can use a frame difference method (inter-frame difference method) to calculate the absolute value of the brightness difference between the current frame image and the previous frame or previous frames of historical images based on the local video image of the current frame and its historical images. It then performs a binarization process, setting the pixels whose absolute value of the brightness difference is greater than the threshold as the first pixel value (such as 255), that is, the pixels of the moving target, and setting the other pixels as the second pixel value (such as 0), to obtain a binary differential image. The server can then perform a connected domain analysis on the differential image to determine the connected area consisting of the target pixels with the first pixel value. One connected area corresponds to one moving target. The connected domain analysis method can use a seed filling method, a two-pass algorithm, or other improved algorithms.

[0071] Afterwards, the server can use each connected area as a candidate detection area for a moving target, or use the minimum circumscribed rectangular area of ​​the connected area as a candidate detection area for a moving target. The server can then determine whether the size (number of pixels or pixel ratio) of each candidate detection area meets the small target size condition, determine the candidate detection area that meets the small target size condition as the target detection area, and obtain the position information of each target detection area in the local video image of the current frame. It is understandable that the moving target in the target detection area may be a target object of the monitoring type, but the type of the target object cannot be determined at this time, or it may be other moving objects of the non-monitoring type.

[0072] In step 204 , the image of the target detection area in the local video image of the current frame is input into a second target detection model for target detection to obtain a second detection result.

[0073] The size of the input image of the second target detection model (such as image resolution 128*128 or 64*64, etc.) is smaller than the size of the input image of the first target detection model (such as 640*640, 640*360, etc.). Specifically, it can be set according to the number of pixels or size occupied by the small-sized target in the local video image. The second target detection model can be established using target detection algorithms such as YOLO and Fast-RCNN, and is trained using sample images of corresponding size and with a large number of pixels of the target object in the sample image.

[0074] The server can capture an image of the target detection area from the local video image of the current frame and input it into the second target detection model for target detection. The output result of the second target detection model can be the type and position information of the target object contained in the target detection area (such as the coordinates (x, y; w, h) of the rectangular detection box), and then the position information can be converted into local position information in the coordinate system of the local video image of the current frame, and the type and local position information of the target object are used as the second detection result. It can be understood that if the size of the target detection area does not meet the size of the input image of the second target detection model, the image of the target detection area can be scaled (resized) to the required size before being input into the second target detection model for target detection.

[0075] Step 205 : obtaining an object detection result of the fused video image of the current frame of the fused video based on the first detection result and the second detection result.

[0076] The fused video is obtained by stitching and fusing the local videos. A mapping relationship exists between the coordinate systems of the local videos and the coordinate system of the fused video. Specifically, a homography transformation matrix can be established to perform coordinate transformation. The server can perform coordinate transformation on the local position information of the target objects detected in steps 202 and 204 based on the coordinate mapping relationship between the local videos and the fused video, obtaining the fused position information of each target object in the coordinate system of the current frame fused video image. For example, the coordinates of the upper left and lower right corners of the rectangular detection frame of the target object are transformed to locate the position information of the rectangular detection frame of the target object in the fused video image as the fused position information of the target object in the coordinate system of the current frame fused video image. The server can then use the type of each target object detected in the first detection result and the second detection result and the converted fused position information, i.e., the combination of the two detection results, as the target detection result of the current frame fused video image. The server can send the target detection result and the fused video together to the display terminal, so that the display terminal can display the fused video and the target detection frame, making it easier for users to view the monitoring video and intelligent monitoring results.

[0077] The above-mentioned target detection method for billion-pixel fusion video reduces the local video image and inputs it into the first target detection model for target detection, thereby realizing rapid detection of larger-sized targets, and performing motion target detection on the local video image to determine the candidate detection area where the moving target is located, and inputs the image of the target detection area that meets the small target size condition into the second target detection model to realize accurate detection of smaller targets. Moreover, since the image size of the target detection area is small, the model needs to process a small amount of data as the input of the second target detection model, so the detection efficiency is high, and then the two types of detection results are merged to obtain the target detection result of the billion-pixel fusion video. In this way, the technical effect of high accuracy and high detection efficiency of target detection for billion-pixel fusion video can be achieved, and computing resources can be saved and hardware computing power requirements can be reduced.

[0078] In addition, since small targets account for a small proportion of pixels in billion-pixel fusion videos and local video images, they contain too little color information such as RGB and discriminative features. Even if small target detection is performed directly on uncompressed billion-pixel fusion videos or local video images, the false detection rate is still high. This method uses motion target detection and detection area size condition screening to preliminarily determine the target detection area where small targets may be located, and then uses the image of this target detection area for small target detection. Since small targets account for a large proportion of pixels in this target detection area, that is, the image of the target detection area mainly contains the characteristic information of small targets, compared to detecting small targets in a whole large image, this method can further improve the accuracy of target detection.

[0079] In another embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame, and the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame, such as Figure 3 As shown, the process of obtaining the target detection result in step 205 specifically includes the following steps:

[0080] Step 301 : performing coordinate transformation on the local position information of the first target object and the second target object according to the coordinate mapping relationship between each local video and the fused video, and obtaining the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame.

[0081] Step 302: Find the first target object and the second target object that meet the fusion position intersection condition based on the fusion position information of the first target object and the second target object, and determine whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, merge the fusion position information of the first target object and the second target object that meet the fusion position intersection condition as the fusion position information of the target merged object.

[0082] Among them, the fusion position intersection condition can be that the intersection ratio of the detection frames of the first target object and the second target object is greater than a threshold, that is, if the detection frame of the first target object detected in the first detection result in the fused video image (fusion position information) intersects with the detection frame of the second target object detected in the second detection result in the fused video image, and the area intersection ratio is greater than the threshold, then the first target object and the second target object can be determined as the same target (target merged object), and the detection frames of the two need to be merged, such as using the minimum circumscribed rectangular frame of the detection frames of the two as the detection frame of the target merged object.

[0083] In step 303, the first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

[0084] Since the same target may be detected simultaneously in the first and second detection results, this embodiment determines the overlap of the fusion positions of the target objects in the detection results of the reduced local video image and the detection results of the motion target detection area image that meets the small target size condition. If the overlap is large (meets the fusion position intersection condition) and the detection type is the same, it means that they are the same target object, and the two are merged. Finally, the independent targets that do not overlap or have a small overlap and the detection type and fusion position information of the merged target are used as the target detection result of the current frame fusion video image, thereby improving the accuracy of the target detection result.

[0085] In another embodiment, the first target object and the second target object that meet the fusion position intersection condition but are of different types may have different detection types for the same target object. Therefore, in order to further improve the detection accuracy, the process of obtaining the target detection result in step 303 specifically includes the following steps:

[0086] If there are a first target object and a second target object of different types that meet the fusion position intersection condition, it is further determined whether the image size difference between the first target object and the second target object is greater than a threshold. The image size difference can be an area ratio (large area divided by small area) or an area difference (large area minus small area). If the image size difference is not greater than the threshold, that is, the areas occupied by the two in the fused video image are equivalent, the first target object and the second target object are merged as the target merged object, and the detection type of the second target object is used as the detection type of the target merged object.

[0087] Then, the first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types and whose image size difference is greater than the threshold, and the target merged object are determined as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

[0088] In this embodiment, if the detection frames of the target object detected in the first detection result and the target object detected in the second detection result in the fused image have a large overlap, but the detection types are different, the difference in the detection frame sizes between the two can be further determined. If the detection frame sizes are comparable, since the second detection result is based on the target detection area where the small target is located, it indicates that the first detection result detected a small target. Therefore, the detection type of the second detection result with higher detection accuracy shall prevail. If the detection frame sizes of the two are significantly different, it is possible that the two target objects overlap, such as if the image of a small target is completely contained in the image of a large target. Therefore, the two targets are independent targets and the detection results must be retained.

[0089] In another embodiment, if Figure 4 As shown, the process of obtaining the target detection result in step 205 specifically includes the following steps:

[0090] In step 401 , the size of the fused video image of the current frame of the fused video is reduced and the image is input into a third target detection model for target detection to obtain a third detection result.

[0091] The third detection result includes the type of the detected third target object and fusion position information of the third target object in the coordinate system of the fused video image of the current frame.

[0092] The input image size of the third target detection model is larger than the input image size of the second target detection model, and may be larger than or equal to the input image size of the first target detection model. Optionally, the third target detection model is the same model as the first target detection model. That is, the fused video image can be reduced to the target size (consistent with the size of the reduced local video image), and then input into the first target detection model for target detection, outputting the position information and detection type of the detected target object in the reduced fused video image, and then converting its coordinates to the fused position information in the coordinate system of the fused video image before compression to obtain the third detection result.

[0093] Step 402: Obtain an object detection result of the current frame fused video image based on the first detection result, the second detection result, and the third detection result.

[0094] The merging process of the first detection result, the second detection result, and the third detection result can be similar to steps 301 to 303 above. The detection frames of the first target object, the second target object, and the third target object that meet the fusion position intersection condition (large overlap) and have the same detection type are merged (such as the minimum bounding rectangle) to serve as the detection frame of the target merged object. Then, the first target object, the second target object, and the third target object that do not meet the fusion position intersection condition or meet the fusion position intersection condition but are of different types, as well as the target merged object, are taken as the final target object detected in the current frame fusion video image, thereby obtaining the target detection result of the current frame fusion video image.

[0095] In practice, if the target scene contains a large surveillance object (such as an aircraft close to an airport when monitoring an airport), multiple sub-cameras may each capture a portion of the large target. If one of the partial video images only contains a small portion of the large target, the target may not be recognized during target detection in the partial video image. The final detection result for the target will be based on the detection result of the image containing the larger portion of the target, resulting in the detection box of the target not containing the entire image of the target, and the detection result accuracy is low. This embodiment further improves the accuracy of target detection results by reducing the fused video before performing target detection and then merging the three types of detection results to obtain the final result.

[0096] In one embodiment, Figure 5 As shown, in step 203, the process of using the frame difference method to detect moving targets in the local video and obtaining the candidate detection areas where each moving target is located in the local video image of the current frame specifically includes the following steps:

[0097] Step 501 : For each channel of partial video, a binary differential image corresponding to the current frame partial video image is obtained based on the current frame partial video image and the historical frame partial video images of the partial video.

[0098] Step 502 , performing connected domain analysis on the differential image to determine connected areas formed by target pixels, and determining candidate detection areas where each moving target is located based on each connected area; the target pixel is the pixel corresponding to the moving target.

[0099] In one implementation, the process of obtaining the differential image in step 501 specifically includes the following steps:

[0100] Step 5011: determine the historical frame local video image that is separated from the current frame local video image by a preset number of frames in the local video as the reference frame image, and determine the historical frame local video images between the reference frame image and the current frame local video image, as well as the current frame local video image, as comparison frame images.

[0101] The server can store the local video image of the current frame (e.g., Y n A certain historical frame local video image before the frame) is used as the reference frame image, such as Y0 frame (the interval between the two is n-1 frames), and Y1 to Y n A total of n frames of local video images are used as comparison frame images.

[0102] Step 5012 , performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial difference images, and performing binarization processing on each initial difference image to obtain a plurality of binarized initial difference images.

[0103] In practice, the server can be Y1 to Y n For each comparison frame image, an initial differential image is calculated with the reference frame image Y0. For example, the brightness difference is calculated pixel by pixel as the brightness value of each pixel in the differential image. Then, the value of the pixel whose brightness value is greater than the preset threshold is set to the first pixel value (such as 255), and the value of the other pixel points is set to the second pixel value (such as 0), thereby obtaining multiple binary initial differential images.

[0104] Step 5013 , the binarized initial differential images are superimposed, and the superimposed images are subjected to morphological processing to obtain a binarized differential image corresponding to the local video image of the current frame.

[0105] In implementation, the server can superimpose multiple initial differential images to obtain a binary differential image corresponding to the local video image of the current frame. Specifically, if the value of the pixel point after superposition is greater than the second pixel value (0), it is set to the first pixel value (255); if it is equal to the second pixel value after superposition, it is set to the second pixel value (0). That is, if the pixel point has the first pixel value in any of the initial differential images, it will remain the first pixel value in the superimposed differential image. Morphological processing can be an opening operation, an erosion operation, an expansion operation, etc., the purpose of which is to reduce noise and interference in the image.

[0106] In this embodiment, since the frame difference method is prone to introducing a large amount of noise when detecting moving targets, morphological processing can usually be performed before performing connected domain analysis on the differential image to filter the image noise and reduce interference. However, since small-sized targets occupy fewer pixels, small-sized targets are easily mistakenly filtered out after performing image morphological processing for noise reduction. Therefore, this method performs differential calculations on multiple comparison frame images and reference frame images. For small-sized targets of monitoring interest, they usually move continuously. Therefore, superimposing multiple frames of initial differential images can enhance the small-sized targets. For noise or moving objects that are not of interest, such as leaves blown by the wind, their range of movement is limited, and the number of pixels they occupy will not increase significantly after superposition. Therefore, denoising and filtering the superimposed differential images can effectively remove noise and retain the pixels of the small-sized targets of interest, thereby improving the efficiency of target detection.

[0107] In one implementation, the connected domain analysis method used in step 502 specifically includes the following steps:

[0108] Step 5021: Use the initial serial number of each target pixel in the differential image as the current serial number for the first updating process.

[0109] The initial sequence number is a sequence number pre-assigned to each target pixel in the differential image in ascending or descending order. The initial sequence number can be obtained based on the coordinates of each pixel. For example, the image coordinates of pixel P are (r, c), where r represents the row number of the pixel and c represents the column number. Then, its initial sequence number is P(i) = (r-1) × w + (c-1), which can achieve ascending order. In other implementations, other methods can also be used to arrange in descending order, which is not limited in this embodiment.

[0110] Step 5022: For each target pixel point, search for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of the target pixel point in the differential image as the first associated pixel points, and use the minimum or maximum current serial number of the target pixel point and each first associated pixel point as the target serial number corresponding to the target pixel point.

[0111] Wherein, adjacent pixel points can be pixel points of 8 neighborhoods, and the corresponding extension direction is 8 directions in total, namely, the upper, lower, left, right, upper left, upper right, lower left, and lower right of the target pixel point. For example, for pixel point Pi, if the right adjacent pixel point P(i+1) is the target pixel point (pixel is 255), it is used as the first associated pixel point, and the search is continued along this direction. If the pixel value of P(i+2) searched is the target pixel point, and P(i+3) is a non-target pixel point (pixel value is 0), then the first associated pixel points searched in this direction are P(i+1) and P(i+2), and the same applies to other directions. It should be noted that, even if P(i+4) is the target pixel point, it is not used as the first associated pixel point because it is not connected to Pi. Then, the largest or smallest sequence number among the current sequence numbers of the target pixel point Pi and each first associated pixel point is used as the target sequence number of the target pixel point Pi. It should be noted that when determining the target sequence number of each target pixel, the rules used must be consistent, such as using the smallest sequence number as the target sequence number, or using the largest sequence number as the target sequence number.

[0112] Step 5023, determining whether the target sequence number of the target pixel point is consistent with the initial sequence number; if the target sequence number of the target pixel point is consistent with the initial sequence number, determining the target sequence number as the updated sequence number of the target pixel point.

[0113] After the server determines the target sequence number for each target pixel, it can compare the target sequence number with the initial sequence number. If the two are consistent, the target sequence number is determined as the updated sequence number of the target pixel. That is, for target pixels with the same target sequence number as the initial sequence number, the target sequence number is used as the updated sequence number of the target pixel.

[0114] Step 5024: If the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence numbers are consistent with the target sequence number of the target pixel point are used as second associated pixels, and the update sequence number of the target pixel point is determined based on the target sequence number and the initial sequence number of the second associated pixel point.

[0115] In implementation, for target pixel points whose target sequence number is inconsistent with the initial sequence number, the server can use other target pixel points whose initial sequence number is consistent with the target sequence number as second associated pixel points, and determine the update sequence number of the target pixel point based on the target sequence number and initial sequence number of the second associated pixel point.

[0116] In one implementation, the server can compare whether the target sequence number and the initial sequence number of the second associated pixel point are the same. If they are the same, the target sequence number of the second associated pixel point is used as the updated sequence number of the target pixel point. If they are not the same, other target pixel points whose initial sequence numbers are consistent with the target sequence number of the second associated pixel point are used as new second associated pixel points, and the server returns to execute the step of comparing whether the target sequence number and the initial sequence number of the second associated pixel point are the same, until a new second associated pixel point with the same target sequence number and initial sequence number is found, that is, the updated sequence number of the target pixel point is determined. During the next update process, the current sequence number of the target pixel point can be updated to the updated sequence number. Optionally, the current sequence numbers of all second associated pixel points involved in the search process can be updated to the same updated sequence number as the target pixel point to improve the update efficiency.

[0117] Step 5025, updates the current serial number of each target pixel point to the update serial number, returns to execute the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image, until the preset number of update processing times is reached, and uses the update serial number determined after the last update processing as the matching serial number, and determines the area composed of target pixel points with the same matching serial number as a connected area.

[0118] Among them, steps 5022 to 5024 can be executed in a loop, and each execution of a loop is an update process. The number of loops can be set, preferably set to 2 times. Thus, the value of the area where each moving target is located becomes the minimum serial number value or the maximum serial number value of its coverage area. If there are k targets, there will be k areas, and the value of each area is consistent, but the values ​​of different areas are different. Therefore, the area composed of target pixels with the same matching serial number can be determined as a connected area where a moving target is located. Optionally, the minimum circumscribed rectangle of the area where the target pixel points with the same matching serial number are located can be used as a connected area. Specifically, the position coordinates (c, r) are placed in different arrays D respectively, which will form k arrays D, D1, D2-D k , each D contains multiple position coordinate information. For example, D1=[(c0,r0),(c1,r1),(c2,r2)]. Calculate the minimum value (x min ,y min ) and the maximum value (x max ,y max ). The calculation formula is as follows:

[0119] x min =MIN(D[i][0])y min =MIN(D[i][1])

[0120] x max=MAX(D[i][0])y max =MAX(D[i][1])

[0121] Where i is the index of all positions in the traversal array D. The obtained (x min ,y min ) and (x max ,y max ) are the coordinates of the upper left and lower right corners of the connected area rectangle of a moving target, which can be used to locate the position of the connected area.

[0122] In one embodiment, after the current sequence number of the target pixel is updated to the updated sequence number in step 5025, the following step is further included: marking the target pixel as a processed pixel. Accordingly, the process of comparing whether the target sequence number and the initial sequence number of the second associated pixel are the same specifically includes the following steps: determining whether the second associated pixel is currently marked as a processed pixel; if the second associated pixel is already marked as a processed pixel, using the updated sequence number corresponding to the second associated pixel as the updated sequence number of the target pixel; if the second associated pixel is not marked as a processed pixel, comparing whether the target sequence number and the initial sequence number of the second associated pixel are the same.

[0123] In one embodiment, the process of using the target sequence number of the second associated pixel point having the same target sequence number and initial sequence number as the update sequence number of the target pixel point specifically includes the following steps: using the target sequence number of the second associated pixel point having the same target sequence number and initial sequence number as the update sequence number, updating the current sequence numbers of the target pixel point and the second associated pixel point to the update sequence number, and marking the target pixel point and the second associated pixel point as processed pixels. Accordingly, the process of determining whether the target sequence number and the initial sequence number of the target pixel point are consistent specifically includes the following steps: determining whether the target pixel point is currently marked as a processed pixel point, and if the target pixel point is not marked as a processed pixel point, determining whether the target sequence number and the initial sequence number of the target pixel point are consistent.

[0124] In this embodiment, since the process of determining the update sequence number of each target pixel point can be processed in parallel, and in the process of determining the update sequence number of the target pixel point, the update sequence number of the second associated pixel point (also the target pixel point) involved can be determined at the same time, therefore, after determining the update sequence numbers of the target pixel point and the second associated pixel point, they can be marked as processed pixels, and then when updating the sequence number of each target pixel point, it can be determined first whether the target pixel point or the second associated pixel point has been processed, thereby improving the efficiency of connected domain analysis.

[0125] Compared with the seed filling method, the two-pass algorithm, etc., the above-mentioned connected domain analysis method has a higher degree of parallelism and can improve the efficiency of connected domain analysis and target detection.

[0126] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiments of the present application also provide a device for detecting objects in billion-pixel fusion videos for implementing the method for detecting objects in billion-pixel fusion videos mentioned above. The solution to the problem provided by this device is similar to the solution described in the method above. Therefore, the specific limitations in the embodiments of the device for detecting objects in one or more billion-pixel fusion videos provided below can be found in the limitations of the method for detecting objects in billion-pixel fusion videos mentioned above, and will not be repeated here.

[0128] In one embodiment, Figure 6 As shown, a target detection device 600 for billion-pixel fusion video is provided, comprising: an acquisition module 601, a first detection module 602, a second detection module 603, and a third detection module 604, wherein:

[0129] The acquisition module 601 is used to acquire multiple channels of local videos captured by the array camera.

[0130] The first detection module 602 is used to reduce the size of the current frame local video image of each of the local videos, and input the reduced-size current frame local video image into the first target detection model for target detection to obtain a first detection result.

[0131] The second detection module 603 is used to perform motion target detection on the local video using a frame difference method, obtain candidate detection areas where each moving target is located in the local video image of the current frame, and determine a target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into the second target detection model for target detection to obtain a second detection result.

[0132] The third detection module 604 is used to obtain the target detection result of the fused video image of the current frame of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusing the multiple local videos.

[0133] In one embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame. The third detection module 604 is specifically used to: perform coordinate conversion on the local position information of the first target object and the second target object according to the coordinate mapping relationship between each of the local videos and the fused video, and obtain the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame; find out the first target object and the second target object that meet the fused position intersection condition according to the fused position information of the first target object and the second target object, and determine whether the types of the first target object and the second target object that meet the fused position intersection condition are the same; if they are the same, merge the fused position information of the first target object and the second target object that meet the fused position intersection condition as the fused position information of the target merged object; determine the first target object and the second target object that do not meet the fused position intersection condition, the first target object and the second target object that meet the fused position intersection condition but are of different types, and the target merged object as the final target object detected in the fused video image of the current frame, and obtain the target detection result of the fused video image of the current frame.

[0134] In one embodiment, the third detection module 604 is specifically used to: reduce the size of the current frame fused video image of the fused video and input it into the third target detection model for target detection to obtain a third detection result; the third detection result includes the type of the detected third target object and the fusion position information of the third target object in the coordinate system of the current frame fused video image; based on the first detection result, the second detection result and the third detection result, obtain the target detection result of the current frame fused video image.

[0135] In one embodiment, the second detection module 603 is specifically used to: for each of the local videos, obtain a binary differential image corresponding to the current frame local video image based on the current frame local video image and the historical frame local video image of the local video; perform connected domain analysis on the differential image to determine the connected area composed of target pixel points, and determine the candidate detection area where each moving target is located based on each connected area; the target pixel point is the pixel point corresponding to the moving target.

[0136] In one embodiment, the second detection module 603 is specifically used to: determine a historical frame local video image in the local video that is separated from the current frame local video image by a preset number of frames as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image; perform pixel-by-pixel difference calculation on each frame of the comparison frame image and the reference frame image to obtain multiple initial differential images, and binarize each of the initial differential images to obtain multiple binarized initial differential images; superimpose each of the binarized initial differential images, and perform morphological processing on the superimposed images to obtain a binarized differential image corresponding to the current frame local video image.

[0137] In one embodiment, the second detection module 603 is specifically used to: use the initial serial number of each target pixel point in the differential image as the current serial number of the first update processing, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; for each target pixel point, along the extension direction of the adjacent pixel points of each target pixel point in the differential image, search for other target pixel points connected in each extension direction as the first associated pixel points, and use the minimum or maximum current serial number of the target pixel point and the current serial number of each first associated pixel point as the target serial number of the target pixel point; determine whether the target serial number of the target pixel point is consistent with the initial serial number; if the target serial number of the target pixel point is consistent with the initial serial number, determine the target serial number as The update sequence number of the target pixel point; if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixel points whose initial sequence number is consistent with the target sequence number of the target pixel point are used as second associated pixel points, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; the current sequence number of each target pixel point is updated to the update sequence number, and the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image is returned to execute for each target pixel point until the preset number of update processing times is reached, the update sequence number determined after the last update processing is used as the matching sequence number, and the area composed of the target pixel points with the same matching sequence number is determined as a connected area.

[0138] In one embodiment, the second detection module 603 is specifically used to: compare whether the target serial number and the initial serial number of the second associated pixel point are the same; if they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, use other target pixel points whose initial serial number is consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to execute the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

[0139] Each module in the above-mentioned object detection device for billion-pixel fusion video can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned target detection method for billion-level pixel fusion video. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a target detection method for billion-level pixel fusion video is implemented.

[0141] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0142] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0146] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for object detection in billion-pixel fusion video, characterized in that: The method comprises: Obtain multiple local videos captured by array cameras; Resizing the local video image of the current frame of each of the local videos, and inputting the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; Performing moving target detection on the local video using a frame difference method to obtain candidate detection regions where each moving target is located in the local video image of the current frame, and determining a target detection region that meets a small target size condition from each candidate detection region; Inputting the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result; Based on the first detection result and the second detection result, obtaining a target detection result of a fused video image of a current frame of the fused video; the fused video is obtained by splicing and fusing the multiple local videos; The method of performing moving target detection on the local video using a frame difference method to obtain candidate detection areas where each moving target is located in the local video image of the current frame includes: For each of the partial videos, obtaining a binary differential image corresponding to the current frame partial video image based on the current frame partial video image and the historical frame partial video images of the partial video; Performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points, and determining the candidate detection area where each moving target is located based on each connected area; the target pixel points are the pixel points corresponding to the moving target; wherein, performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points includes: using the initial serial number of each target pixel point in the differential image as the current serial number of the first update processing, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; for each target pixel point, searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image as the first associated pixel point, and using the minimum or maximum current serial number of the target pixel point and the current serial number of each first associated pixel point as the target serial number of the target pixel point; judging the target pixel point whether the target sequence number and the initial sequence number are consistent; if the target sequence number and the initial sequence number of the target pixel point are consistent, the target sequence number is determined as the update sequence number of the target pixel point; if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels with the initial sequence number consistent with the target sequence number of the target pixel point are used as second associated pixel points, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; the current sequence number of each target pixel point is updated to the update sequence number, and the step of searching for other target pixels connected in each extension direction along the extension direction of the adjacent pixels of each target pixel point in the differential image is returned to execute for each target pixel point, until the preset number of update processing times is reached, the update sequence number determined after the last update processing is used as the matching sequence number, and the area composed of the target pixels with the same matching sequence number is determined as a connected area.

2. The method according to claim 1, characterized in that The first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame; The obtaining, based on the first detection result and the second detection result, a target detection result of the fused video image of the current frame of the fused video, includes: Performing coordinate conversion on the local position information of the first target object and the second target object according to the coordinate mapping relationship between each of the partial videos and the fused video to obtain fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame; Searching, based on the fusion position information of the first target object and the second target object, for a first target object and a second target object that meet a fusion position intersection condition, and determining whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, merging the fusion position information of the first target object and the second target object that meet the fusion position intersection condition as the fusion position information of the target merging object; The first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

3. The method according to claim 1, characterized in that The obtaining, based on the first detection result and the second detection result, a target detection result of the fused video image of the current frame of the fused video, includes: reducing the size of the fused video image of the current frame of the fused video and inputting the image into a third target detection model for target detection, obtaining a third detection result; the third detection result includes the type of the detected third target object and fused position information of the third target object in the coordinate system of the fused video image of the current frame; An object detection result of the current frame fused video image is obtained based on the first detection result, the second detection result, and the third detection result.

4. The method according to claim 1, wherein The step of obtaining a binary differential image corresponding to the local video image of the current frame based on the local video image of the current frame and the local video image of the historical frame includes: Determine a historical frame local video image that is a preset number of frames away from the current frame local video image in the partial video as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image; Performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial differential images, and performing binarization processing on each of the initial differential images to obtain a plurality of binarized initial differential images; The binarized initial differential images are superimposed, and the superimposed images are morphologically processed to obtain a binarized differential image corresponding to the local video image of the current frame.

5. The method according to claim 1, characterized in that The step of determining the update sequence number of the target pixel point according to the target sequence number and the initial sequence number of the second associated pixel point includes: Compare whether the target serial number and the initial serial number of the second associated pixel point are the same. If they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, use other target pixel points whose initial serial number is consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

6. An object detection device for billion-pixel fusion video, characterized in that: The device comprises: An acquisition module is used to acquire multiple local videos captured by array cameras; A first detection module is configured to reduce the size of a local video image of a current frame of each of the local videos, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; The second detection module is configured to perform motion target detection on the local video using a frame difference method to obtain candidate detection areas where each motion target is located in the local video image of the current frame, and determine a target detection area that meets a small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into a second target detection model to perform target detection to obtain a second detection result; wherein, performing motion target detection on the local video using a frame difference method to obtain candidate detection areas where each motion target is located in the local video image of the current frame includes: For each of the partial videos, obtaining a binary differential image corresponding to the current frame partial video image based on the current frame partial video image and the historical frame partial video images of the partial video; Performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points, and determining the candidate detection area where each moving target is located based on each connected area; the target pixel points are the pixel points corresponding to the moving target; wherein, performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points includes: using the initial serial number of each target pixel point in the differential image as the current serial number of the first update processing, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; for each target pixel point, searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image as the first associated pixel point, and using the minimum or maximum current serial number of the target pixel point and the current serial number of each first associated pixel point as the target serial number of the target pixel point; judging the target pixel point whether the target sequence number and the initial sequence number of the target pixel point are consistent; if the target sequence number and the initial sequence number of the target pixel point are consistent, the target sequence number is determined as the update sequence number of the target pixel point; if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence number is consistent with the target sequence number of the target pixel point are used as second associated pixel points, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; the current sequence number of each target pixel point is updated to the update sequence number, and the step of searching for other target pixels connected in each extension direction along the extension direction of the adjacent pixels of each target pixel point in the differential image is returned to be executed for each target pixel point, until a preset number of update processing times is reached, the update sequence number determined after the last update processing is used as the matching sequence number, and the area formed by the target pixels with the same matching sequence number is determined as a connected area; The third detection module is used to obtain the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusion of the multiple local videos.

7. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system includes an array camera, a server, and a display terminal. The array camera includes multiple sub-cameras, wherein: The array camera is used to capture a target scene through multiple sub-cameras and send the captured multiple local videos to a server; The server is used to splice and fuse the acquired partial videos to obtain a fused video; The server is further configured to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; perform motion target detection on the local video using a frame difference method to obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine a target detection area that meets a small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result; and obtain a target detection result of the fused video image of the current frame of the fused video based on the first detection result and the second detection result; The method of performing moving target detection on the local video using a frame difference method to obtain candidate detection areas where each moving target is located in the local video image of the current frame includes: For each of the partial videos, obtaining a binary differential image corresponding to the current frame partial video image based on the current frame partial video image and the historical frame partial video images of the partial video; Performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points, and determining the candidate detection area where each moving target is located based on each connected area; the target pixel points are the pixel points corresponding to the moving target; wherein, performing a connected domain analysis on the differential image to determine the connected area formed by the target pixel points includes: using the initial serial number of each target pixel point in the differential image as the current serial number of the first update processing, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; for each target pixel point, searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image as the first associated pixel point, and using the minimum or maximum current serial number of the target pixel point and the current serial number of each first associated pixel point as the target serial number of the target pixel point; judging the target pixel point whether the target sequence number and the initial sequence number of the target pixel point are consistent; if the target sequence number and the initial sequence number of the target pixel point are consistent, the target sequence number is determined as the update sequence number of the target pixel point; if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence number is consistent with the target sequence number of the target pixel point are used as second associated pixel points, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; the current sequence number of each target pixel point is updated to the update sequence number, and the step of searching for other target pixels connected in each extension direction along the extension direction of the adjacent pixels of each target pixel point in the differential image is returned to be executed for each target pixel point, until a preset number of update processing times is reached, the update sequence number determined after the last update processing is used as the matching sequence number, and the area formed by the target pixels with the same matching sequence number is determined as a connected area; The server is further configured to send the fused video and the target detection result to the display terminal.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images