Target detection method, device and system for hundred million-level pixel fusion video

Abstract

The invention relates to a target detection method and device for a hundred-million-level pixel fusion video, a hundred-million-level pixel calculation imaging system, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring multiple local videos shot by an array camera; performing size reduction processing on the local video image of the current frame, and inputting the local video image into a first target detection model for target detection to obtain a first detection result; performing moving target detection on the local video by adopting a frame difference method, and determining a target detection area meeting a small target size condition; inputting the image of the target detection area into a second target detection model for target detection to obtain a second detection result; and obtaining a target detection result of the current frame fusion video image of the fusion video based on the first detection result and the second detection result. By adopting the method, the target detection accuracy and detection efficiency of the hundred million-level pixel fusion video can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for detecting a target in billion-pixel fusion video, a billion-pixel computational imaging system, a computer device, a storage medium, and a computer program product. Background Art

[0002] Video surveillance has been widely used in the security field due to its advantages of high reliability, strong timeliness, and easy viewing. When the monitoring range is large, in order to achieve ultra-high-definition video monitoring of large scenes, a billion-pixel computational imaging system based on array cameras has emerged. The array camera contains multiple long-focus and narrow-field-of-view sub-cameras to shoot multiple local high-definition videos of the monitoring scene, and then the local videos are spliced ​​and fused to obtain ultra-high-definition fused videos with a large or wide field of view up to the billion-pixel level. It is suitable for video shooting and security monitoring of large scenes such as airports, highways, parks, sports stadiums, borders, and sea surfaces.

[0003] In related technologies, billion-pixel computational imaging systems can also provide intelligent target detection functions, and detect monitoring targets in fused video images in real time through target detection models, such as people, cars, airplanes, birds, drones, etc. Due to the large amount of data in billion-pixel fused videos, if target detection is performed directly on billion-pixel fused video images or all local video images, the detection efficiency is low, and the hardware computing power requirements and computing resource consumption are high. Therefore, in actual applications, in order to improve target detection efficiency and save computing resources, the size of each local video image is usually reduced, such as downsampling (downsampling, reducing) an image with a resolution of 3840*2160 to a resolution of 640*360, and then input into the target detection model for detection.

[0004] However, the detection accuracy of the method of reducing the size of the local video image before performing target detection will be greatly reduced, especially for small targets with a small pixel ratio such as drones and flying birds, the missed detection rate or false detection rate is high. Therefore, this method is difficult to achieve both high detection accuracy and high detection efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a target detection method, device, computer equipment, computer-readable storage medium and computer program product that can improve the target detection accuracy and detection efficiency of billion-pixel fusion videos in response to the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a method for object detection in a billion-pixel fusion video. The method comprises:

[0007] Acquire multiple local videos captured by array cameras;

[0008] Reducing the size of the local video image of the current frame of each local video, and inputting the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result;

[0009] Using a frame difference method to perform moving target detection on the local video, obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine a target detection area that meets a small target size condition from each of the candidate detection areas;

[0010] Inputting the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result;

[0011] Based on the first detection result and the second detection result, a target detection result of a fused video image of a current frame of the fused video is obtained; the fused video is obtained based on the splicing and fusion of the multiple local videos.

[0012] In one embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame;

[0013] The obtaining, based on the first detection result and the second detection result, a target detection result of the current frame fused video image of the fused video includes:

[0014] According to the coordinate mapping relationship between each of the local videos and the fused video, coordinate transformation is performed on the local position information of the first target object and the second target object to obtain the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame;

[0015] According to the fusion position information of the first target object and the second target object, the first target object and the second target object that meet the fusion position intersection condition are found, and it is determined whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, the fusion position information of the first target object and the second target object that meet the fusion position intersection condition are merged as the fusion position information of the target merged object;

[0016] The first target object and the second target object that do not satisfy the fusion position intersection condition, the first target object and the second target object that satisfy the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fused video image, and the target detection result of the current frame fused video image is obtained.

[0017] In one embodiment, obtaining the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result includes:

[0018] The size of the current frame fused video image of the fused video is reduced and then input into a third target detection model for target detection to obtain a third detection result; the third detection result includes the type of the detected third target object and the fusion position information of the third target object in the coordinate system of the current frame fused video image;

[0019] Based on the first detection result, the second detection result and the third detection result, an object detection result of the current frame fused video image is obtained.

[0020] In one embodiment, the step of performing moving target detection on the local video using a frame difference method to obtain a candidate detection area where each moving target in the local video image of the current frame is located includes:

[0021] For each of the local videos, according to the local video image of the current frame and the local video image of the historical frame, a binary difference image corresponding to the local video image of the current frame is obtained;

[0022] Connected domain analysis is performed on the differential image to determine connected areas formed by target pixel points, and candidate detection areas where each moving target is located are determined based on each connected area; the target pixel point is a pixel point corresponding to the moving target.

[0023] In one embodiment, obtaining a binary difference image corresponding to the local video image of the current frame according to the local video image of the current frame and the local video image of the historical frame includes:

[0024] Determine a historical frame local video image in the local video that is separated from the current frame local video image by a preset number of frames as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image;

[0025] Performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial differential images, and binarizing each of the initial differential images to obtain a plurality of binarized initial differential images;

[0026] The binarized initial differential images are superimposed, and the superimposed images are subjected to morphological processing to obtain a binarized differential image corresponding to the local video image of the current frame.

[0027] In one embodiment, the performing a connected domain analysis on the differential image to determine a connected region formed by target pixels includes:

[0028] Using the initial serial number of each target pixel point in the differential image as the current serial number of the first update process, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image;

[0029] For each of the target pixel points, in the differential image, along the extension direction of the neighboring pixel points of each of the target pixel points, other target pixel points connected in each extension direction are searched as first associated pixel points, and the minimum or maximum current sequence number of the target pixel point and the current sequence numbers of each of the first associated pixel points is used as the target sequence number of the target pixel point;

[0030] Determine whether the target sequence number and the initial sequence number of the target pixel are consistent; if the target sequence number and the initial sequence number of the target pixel are consistent, determine the target sequence number as the update sequence number of the target pixel;

[0031] If the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence numbers are consistent with the target sequence number of the target pixel point are used as second associated pixels, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point;

[0032] The current serial number of each of the target pixel points is updated to the update serial number, and the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each of the target pixel points in the differential image is returned to be executed for each of the target pixel points until a preset number of update processing times is reached, and the update serial number determined after the last update processing is used as the matching serial number, and the area formed by the target pixel points with the same matching serial number is determined as a connected area.

[0033] In one embodiment, determining the update sequence number of the target pixel point according to the target sequence number and the initial sequence number of the second associated pixel point includes:

[0034] Compare whether the target serial number and the initial serial number of the second associated pixel point are the same. If they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, take other target pixel points whose initial serial numbers are consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to execute the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

[0035] In a second aspect, the present application also provides a device for object detection in a billion-pixel fusion video. The device comprises:

[0036] An acquisition module is used to acquire multiple local videos captured by array cameras;

[0037] A first detection module is used to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result;

[0038] The second detection module is used to perform moving target detection on the local video by using a frame difference method, obtain candidate detection areas where each moving target is located in the local video image of the current frame, and determine a target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into a second target detection model for target detection, and obtain a second detection result;

[0039] The third detection module is used to obtain the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusion of the multiple local videos.

[0040] In a third aspect, the present application also provides a billion-pixel computational imaging system. The billion-pixel computational imaging system includes an array camera, a server, and a display terminal, wherein the array camera includes multiple sub-cameras, wherein:

[0041] The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server;

[0042] The server is used for splicing and fusing the acquired local videos to obtain a fused video;

[0043] The server is also used to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame to the first target detection model for target detection to obtain a first detection result; use the frame difference method to perform motion target detection on the local video to obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine the target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame to the second target detection model for target detection to obtain a second detection result; based on the first detection result and the second detection result, obtain the target detection result of the fused video image of the current frame of the fused video;

[0044] The server is further configured to send the fused video and the target detection result to the display terminal.

[0045] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect are implemented.

[0046] In a fifth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0047] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0048] The above-mentioned target detection method, device, billion-pixel computational imaging system, computer equipment, storage medium and computer program product for billion-pixel fusion video achieve rapid detection of larger-sized targets by reducing the local video image and inputting it into the first target detection model for target detection, and perform motion target detection on the local video image to determine the candidate detection area where the moving target is located, and input the image of the target detection area that meets the small target size condition into the second target detection model to achieve accurate detection of smaller targets. Moreover, since the image size of the target detection area is small, it serves as the input of the second target detection model, and the model needs to process a small amount of data, so the detection efficiency is high. The two types of detection results are then merged to obtain the target detection result of the billion-pixel fusion video, thereby achieving the technical effect of both high accuracy and high detection efficiency for target detection of billion-pixel fusion video.

[0049] In addition, since the proportion of small targets in the billion-pixel fusion video and local video images is small, the color information and discriminative features such as RGB contained in them are too small. Even if the uncompressed billion-pixel fusion video or local video image is directly used for small target detection, the false detection rate is still high. This method preliminarily determines the target detection area where the small target may be located through motion target detection and detection area size condition screening, and then uses the image of the target detection area for small target detection. Since the small target accounts for a large proportion of the pixels in the target detection area, that is, the image of the target detection area mainly contains the characteristic information of the small target, compared with detecting small targets in a whole large image, this method can further improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a billion-pixel computational imaging system in an example;

[0051] Figure 2 A schematic diagram of a flow chart of a method for object detection in a billion-pixel fusion video according to an embodiment;

[0052] Figure 3 A schematic diagram of a process for determining a target detection result in one embodiment;

[0053] Figure 4 A schematic diagram of a process for determining a target detection result in another embodiment;

[0054] Figure 5 A schematic diagram of a process for determining a candidate detection area in one embodiment;

[0055] Figure 6 It is a structural block diagram of an object detection device for billion-level pixel fusion video in one embodiment;

[0056] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The object detection method of the billion-pixel fusion video provided in the embodiment of the present application can be applied to Figure 1The billion-pixel computational imaging system shown in FIG. 1 . The array camera 102 and the display terminal 106 communicate with the server 104 through a network respectively. The display terminal 106 can be, but is not limited to, various electronic devices with display units such as personal computers, laptops, smart phones, tablet computers, smart displays, Internet of Things devices, portable wearable devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] In one embodiment, Figure 2 As shown in the figure, a target detection method for billion-pixel fusion video is provided, which can be applied to Figure 1 In this embodiment, the method includes the following steps:

[0060] Step 201: Acquire multiple channels of local videos captured by array cameras.

[0061] Among them, the array camera 102 can adopt the array camera in the prior art or other new array cameras. The present application does not limit the specific structure of the array camera. It includes multiple sub-cameras with different shooting angles, such as M (rows) * N (columns), which respectively shoot local high-definition videos (such as 4K videos) of the target scene. The shooting ranges of adjacent sub-cameras usually have overlapping areas. In some examples, the array camera can also include a panoramic camera. Compared with other sub-cameras that shoot local videos, the panoramic camera is a camera with a short focal length and a wide field of view. Its field of view can cover the field of view of other sub-cameras, which is used to assist in establishing image stitching models and color correction, etc. The array camera can send the captured local videos to the server through the communication module.

[0062] After receiving the local videos from each channel, the server can use the pre-deployed image stitching and fusion model to stitch, fuse, color correct, crop, and process a group of local video images taken by each sub-camera at the same or similar time to obtain an ultra-high-definition fused video image with a large or wide field of view and up to 100 million pixels. Then, the multi-frame fused video image can be generated into a fused video stream and sent to the display terminal for display.

[0063] It is understandable that the partial video may be a video shot in real time or a pre-recorded video, and this application does not limit this.

[0064] Step 202 , downsizing the local video images of the current frames of the local videos of each channel, and inputting the downsized local video images of the current frames into a first target detection model for target detection to obtain a first detection result.

[0065] After receiving the partial video, the server may perform target detection frame by frame, or may perform target detection according to a preset time period (such as 1 second) or at a preset number of frames. For the currently processed partial video image (current frame partial video image), the server may downsample it to reduce the size of the current frame partial video image to the target size.

[0066] Then, the server can input the current frame local video image (reduced image) of the target size into the first target detection model for target detection. The first target detection model can be established using target detection algorithms such as YOLO and Fast-RCNN, and trained using sample images of the target size. The type of target object detected by the first target detection model can be one or more of a person, a car, an airplane, a flying bird, a drone, a flame, thick smoke, etc., which can be set as needed. The output result of the first target detection model can be the type and position information of each target object in the reduced local video image (such as the coordinates (x, y, w, h) of the rectangular detection box), and then the position information can be converted into local position information in the coordinate system of the uncompressed local video image, and the type and local position information of the target object are used as the first detection result.

[0067] It should be noted that the shooting time of the current frame local video image of each local video is the same or similar (there may be a certain exposure time difference). The current frame fused video image of the fused video is obtained by splicing and fusing the current frame local video images of each local video.

[0068] Step 203, using the frame difference method to perform moving target detection on the local video, obtain candidate detection areas where each moving target is located in the local video image of the current frame, and determine the target detection area that meets the small target size condition from each candidate detection area.

[0069] Among them, the small target size condition can be set according to the number of pixels or the pixel ratio of the small target in the local video image. The small target is a target with a smaller pixel ratio in the image relative to other target objects with a larger pixel ratio in the local video image. It can be a target object with a smaller absolute size (such as flying birds, drones, etc.), or other target objects with a longer shooting distance, such as cars and people in the distance, or large airplanes in the sky farther away, so the small target is independent of the type of the target object. The number of pixels less than the first threshold, or the number of pixels after the size reduction processing in step 202 is less than the second threshold, can be used as a small target size condition. The specific threshold can be pre-set in combination with factors such as the distance of the shooting scene, the absolute size of the target object to be monitored, and the reduction ratio of the size reduction processing.

[0070] The server can use the frame difference method (inter-frame difference method) to calculate the absolute value of the brightness difference between the current frame image and the historical frame image based on the local video image of the current frame and its previous frame or previous frames of historical images, and then perform binarization processing, set the pixel points whose absolute value of the brightness difference is greater than the threshold to the first pixel value (such as 255), that is, the pixel point of the moving target, and set the other pixel points to the second pixel value (such as 0), and obtain a binary differential image. Then, the server can perform a connected domain analysis on the differential image to determine the connected area composed of the target pixel points of the first pixel value, and one connected area corresponds to one moving target. Among them, the connected domain analysis method can use a seed filling method, a two-pass algorithm or other improved algorithms.

[0071] Afterwards, the server can use each connected area as a candidate detection area for a moving target, or use the minimum circumscribed rectangular area of ​​the connected area as a candidate detection area for a moving target. Then, the server can determine whether the size (number of pixels or pixel ratio) of each candidate detection area meets the small target size condition, determine the candidate detection area that meets the small target size condition as the target detection area, and obtain the position information of each target detection area in the local video image of the current frame. It can be understood that the moving target in the target detection area may be a target object of the monitoring type, but the type of the target object cannot be determined at this time, and it may also be other moving objects of non-monitoring type.

[0072] Step 204: input the image of the target detection area in the local video image of the current frame into the second target detection model for target detection to obtain a second detection result.

[0073] The size of the input image of the second target detection model (such as image resolution 128*128 or 64*64, etc.) is smaller than the size of the input image of the first target detection model (such as 640*640, 640*360, etc.), and can be set specifically according to the number of pixels or size occupied by the small-sized target in the local video image. The second target detection model can be established using target detection algorithms such as yolo and fast-rcnn, and trained using sample images of corresponding sizes and with a larger number of pixels of the target object in the sample image.

[0074] The server can capture an image of the target detection area from the local video image of the current frame, and input it into the second target detection model for target detection. The output result of the second target detection model can be the type and position information of the target object contained in the target detection area (such as the coordinates (x, y; w, h) of the rectangular detection box), and then the position information can be converted into local position information in the coordinate system of the local video image of the current frame, and the type and local position information of the target object are used as the second detection result. It can be understood that if the size of the target detection area does not meet the size of the input image of the second target detection model, the image of the target detection area can be scaled (resized) to a required size, and then input into the second target detection model for target detection.

[0075] Step 205 , based on the first detection result and the second detection result, obtain the target detection result of the fused video image of the current frame of the fused video.

[0076] Among them, the fused video is obtained by splicing and fusion of each local video, and there is a mapping relationship between the coordinate system of each local video and the coordinate system of the fused video. Specifically, a homography transformation matrix can be established for coordinate transformation. The server can perform coordinate transformation on the local position information of the target object detected in step 202 and step 204 according to the coordinate mapping relationship between each local video and the fused video, and obtain the fusion position information of each target object in the coordinate system of the current frame fusion video image, such as performing coordinate transformation on the coordinates of the two points of the upper left corner and the lower right corner of the rectangular detection frame of the target object, and locating the position information of the rectangular detection frame of the target object in the fusion video image as the fusion position information of the target object in the coordinate system of the current frame fusion video image. Then, the server can use the type of each target object detected in the first detection result and the second detection result and the converted fusion position information, that is, the collection of the two detection results, as the target detection result of the current frame fusion video image. The server can send the target detection result and the fusion video to the display terminal together, so that the display terminal can display the fusion video and the target detection frame, which is convenient for users to view the monitoring video and intelligent monitoring results.

[0077] The above-mentioned target detection method for billion-pixel fusion video achieves rapid detection of larger-sized targets by shrinking the local video image and inputting it into the first target detection model for target detection, and performs motion target detection on the local video image to determine the candidate detection area where the moving target is located, and inputs the image of the target detection area that meets the small target size condition into the second target detection model to achieve accurate detection of smaller targets. Moreover, since the image size of the target detection area is small, the model needs to process a small amount of data as the input of the second target detection model, so the detection efficiency is high, and then the two types of detection results are merged to obtain the target detection result of the billion-pixel fusion video. Thus, the technical effect of high accuracy and high detection efficiency for target detection of billion-pixel fusion video can be achieved, and computing resources can be saved and hardware computing power requirements can be reduced.

[0078] In addition, since the proportion of small targets in the billion-pixel fusion video and local video images is small, the color information and discriminative features such as RGB contained in them are too small. Even if the uncompressed billion-pixel fusion video or local video image is directly used for small target detection, the false detection rate is still high. This method preliminarily determines the target detection area where the small target may be located through motion target detection and detection area size condition screening, and then uses the image of the target detection area for small target detection. Since the small target accounts for a large proportion of the pixels in the target detection area, that is, the image of the target detection area mainly contains the characteristic information of the small target, compared with detecting small targets in a whole large image, this method can further improve the accuracy of target detection.

[0079] In another embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame, and the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame, such as Figure 3 As shown, the process of obtaining the target detection result in step 205 specifically includes the following steps:

[0080] Step 301 , based on the coordinate mapping relationship between each local video and the fused video, coordinate transformation is performed on the local position information of the first target object and the second target object to obtain the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame.

[0081] Step 302, based on the fusion position information of the first target object and the second target object, find out the first target object and the second target object that meet the fusion position intersection condition, and determine whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, merge the fusion position information of the first target object and the second target object that meet the fusion position intersection condition as the fusion position information of the target merged object.

[0082] Among them, the fusion position intersection condition can be that the intersection ratio of the detection frames of the first target object and the second target object is greater than a threshold, that is, if the detection frame of the first target object detected in the first detection result in the fused video image (fusion position information) intersects with the detection frame of the second target object detected in the second detection result in the fused video image, and the area intersection ratio is greater than the threshold, then the first target object and the second target object can be determined as the same target (target merged object), and the detection frames of the two need to be merged, such as using the minimum circumscribed rectangular frame of the detection frames of the two as the detection frame of the target merged object.

[0083] Step 303, the first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

[0084] Since the same target may be detected simultaneously in the first detection result and the second detection result, this embodiment determines the overlap of the fusion positions of the target objects in the detection results of the reduced local video image and the detection results of the moving target detection area image that meets the small target size condition. If the overlap is large (satisfying the fusion position intersection condition) and the detection type is the same, it means that they are the same target object, so the two are merged. Finally, the independent target that does not overlap or has a small overlap and the detection type and fusion position information of the merged target are used as the target detection result of the current frame fusion video image, thereby improving the accuracy of the target detection result.

[0085] In another embodiment, in the first target object and the second target object that meet the fusion position intersection condition but are of different types, there may be a situation where the detection types of the same target object are different. Therefore, in order to further improve the detection accuracy, in step 303, the process of obtaining the target detection result specifically includes the following steps:

[0086] If there are a first target object and a second target object that meet the intersection condition of the fusion position but are of different types, it is further determined whether the image size difference between the first target object and the second target object is greater than a threshold. The image size difference can be an area ratio (large area divided by small area) or an area difference (large area minus small area). If the image size difference is not greater than the threshold, that is, the areas occupied by the two in the fused video image are equivalent, the first target object and the second target object are merged as the target merged object, and the detection type of the second target object is used as the detection type of the target merged object.

[0087] Then, the first target object and the second target object that do not meet the fusion position intersection condition, the first target object and the second target object that meet the fusion position intersection condition but are of different types and whose image size difference is greater than a threshold, and the target merged object are determined as the final target object detected in the current frame fused video image, and the target detection result of the current frame fused video image is obtained.

[0088] In this embodiment, if the detection frame overlap of the target object detected in the first detection result and the target object detected in the second detection result in the fused image is relatively large, but the detection types are different, the difference in the detection frame sizes of the two can be further determined. If the detection frame sizes are comparable, since the second detection result is obtained based on the detection of the target detection area where the small target is located, it means that a small-sized target is detected in the first detection result, and therefore the detection type of the second detection result with higher detection accuracy shall prevail. If the detection frame sizes of the two are quite different, it is possible that the positions of the two target objects overlap, such as the image of a small target is completely contained in the image of a large target, and therefore the two targets are independent targets and the detection results need to be retained.

[0089] In another embodiment, if Figure 4 As shown, the process of obtaining the target detection result in step 205 specifically includes the following steps:

[0090] Step 401, reducing the size of the fused video image of the current frame of the fused video and inputting the image into a third target detection model for target detection to obtain a third detection result.

[0091] The third detection result includes the type of the detected third target object and the fusion position information of the third target object in the coordinate system of the fused video image of the current frame.

[0092] The input image size of the third target detection model is larger than the input image size of the second target detection model, and may be larger than or equal to the input image size of the first target detection model. Optionally, the third target detection model is the same model as the first target detection model. That is, the fused video image can be reduced to a target size (consistent with the size of the reduced local video image), and then input into the first target detection model for target detection, outputting the position information and detection type of the detected target object in the reduced fused video image, and then converting its coordinates to the fused position information in the coordinate system of the fused video image before compression to obtain a third detection result.

[0093] Step 402: Obtain the target detection result of the current frame fused video image based on the first detection result, the second detection result and the third detection result.

[0094] Among them, the merging process of the first detection result, the second detection result and the third detection result can be similar to the above steps 301 to 303, and the detection frames of the target objects that meet the fusion position intersection condition (large overlap) and the same detection type among the detected first target object, second target object and third target object are merged (such as the minimum circumscribed rectangular frame) as the detection frame of the target merged object. Then, the first target object, the second target object and the third target object that do not meet the fusion position intersection condition or meet the fusion position intersection condition but are of different types, as well as the target merged object, are taken as the final target object detected in the current frame fusion video image, and the target detection result of the current frame fusion video image is obtained.

[0095] In practice, if there is a large-sized monitoring object in the target scene (such as an airplane close to the airport when monitoring the airport), there may be multiple sub-cameras that capture a portion of the image of the large target. If one of the local video images only contains a small portion of the image of the large target, the target may not be recognized when the local video image is detected. The final detection result of the target will be based on the detection result containing the larger portion of the target image, resulting in the detection frame of the target not containing the entire image of the target, and the detection result is less accurate. This embodiment performs target detection after reducing the fused video, and then merges the three types of detection results to obtain the final result, which can further improve the accuracy of the target detection result.

[0096] In one embodiment, Figure 5 As shown, in step 203, the process of using the frame difference method to detect moving targets in the local video and obtaining the candidate detection area where each moving target is located in the local video image of the current frame specifically includes the following steps:

[0097] Step 501 : for each channel of local video, a binary difference image corresponding to the local video image of the current frame is obtained according to the local video image of the current frame and the local video image of the historical frame.

[0098] Step 502, performing connected domain analysis on the differential image to determine the connected areas formed by the target pixels, and determining the candidate detection areas where the moving targets are located based on the connected areas; the target pixels are the pixels corresponding to the moving targets.

[0099] In one implementation, the process of obtaining the differential image in step 501 specifically includes the following steps:

[0100] Step 5011, determine the historical frame local video image that is separated from the current frame local video image by a preset number of frames in the local video as the reference frame image, and determine the historical frame local video images between the reference frame image and the current frame local video image, and the current frame local video image as comparison frame images.

[0101] The server can store the local video image of the current frame (such as Y n A certain historical frame local video image before the Y0 frame is taken as the reference frame image, for example, it is recorded as the Y0 frame (the interval between the two is n-1 frames), and Y1 to Y n A total of n frames of local video images are used as comparison frame images.

[0102] Step 5012, performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial differential images, and performing binarization processing on each initial differential image to obtain a plurality of binarized initial differential images.

[0103] In implementation, the server may be Y1 to Y n For each comparison frame image, an initial differential image is calculated with the reference frame image Y0. For example, the brightness difference is calculated pixel by pixel as the brightness value of each pixel in the differential image. Then, the value of the pixel whose brightness value is greater than a preset threshold is set to the first pixel value (such as 255), and the value of other pixels is set to the second pixel value (such as 0), so as to obtain multiple binary initial differential images.

[0104] Step 5013, superimposing the binarized initial differential images, and performing morphological processing on the superimposed images to obtain a binarized differential image corresponding to the local video image of the current frame.

[0105] In implementation, the server can superimpose multiple initial differential images to obtain a binary differential image corresponding to the local video image of the current frame. Specifically, if the value of the pixel point is greater than the second pixel value (0) after superposition, it is set to the first pixel value (255); if it is equal to the second pixel value after superposition, it is set to the second pixel value (0), that is, if the pixel point is the first pixel value in any initial differential image, it will remain the first pixel value in the superimposed differential image. Morphological processing can be an opening operation, an erosion operation, an expansion operation, etc., the purpose of which is to reduce noise and interference in the image.

[0106] In this embodiment, since a large amount of noise is easily introduced when the frame difference method is used to detect moving targets, morphological processing can usually be performed before the connected domain analysis is performed on the differential image to filter the image noise and reduce interference. However, since small-sized targets occupy fewer pixels, it is easy to mistakenly filter out small-sized targets after image morphological processing for noise reduction. Therefore, this method performs differential calculations between multiple frames of comparison frame images and reference frame images. For small-sized targets of concern to monitoring, they usually move continuously. Therefore, multiple frames of initial differential images are superimposed to enhance small-sized targets. For noise or moving objects that are not concerned, such as leaves blown by the wind, their range of movement is limited, and the number of pixels they occupy will not increase significantly after superposition. Therefore, the superimposed differential images are subjected to noise reduction and filtering, which can effectively remove noise and retain the pixels of small-sized targets of concern, thereby improving the efficiency of target detection.

[0107] In one implementation, the connected domain analysis method used in step 502 specifically includes the following steps:

[0108] Step 5021, using the initial serial number of each target pixel in the differential image as the current serial number for the first update process.

[0109] The initial serial number is an ascending or descending serial number pre-assigned to each target pixel in the differential image. The initial serial number can be obtained based on the coordinates of each pixel. For example, the image coordinates of pixel P are (r, c), where r represents the row number of the pixel, and c represents the column number. Then its initial serial number is P(i) = (r-1) × w + (c-1), thereby achieving ascending order. In other implementations, other methods can also be used to arrange in descending order, which is not limited in this embodiment.

[0110] Step 5022, for each target pixel point, search for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of the target pixel point in the differential image as the first associated pixel points, and use the minimum or maximum current serial number of the target pixel point and each first associated pixel point as the target serial number corresponding to the target pixel point.

[0111] Among them, the adjacent pixel points can be the pixel points of 8 neighborhoods, and the corresponding extension direction is the upper, lower, left, right, upper left, upper right, lower left, and lower right directions of the target pixel point, which are 8 directions in total. For example, for pixel point Pi, if the right adjacent pixel point P(i+1) is the target pixel point (the pixel is 255), it is used as the first associated pixel point, and the search continues along this direction. If the pixel value of P(i+2) is searched as the target pixel point, and P(i+3) is a non-target pixel point (the pixel value is 0), then the first associated pixel point searched in this direction is P(i+1) and P(i+2), and the same applies to other directions. It should be noted that even if P(i+4) is the target pixel point, it is not used as the first associated pixel point because it is not connected to Pi. Then, the maximum or minimum sequence number among the current sequence numbers of the target pixel point Pi and each first associated pixel point is used as the target sequence number of the target pixel point Pi. It should be noted that when determining the target serial number of each target pixel, the adopted rules must be consistent, such as using the smallest serial number as the target serial number, or using the largest serial number as the target serial number.

[0112] Step 5023, determining whether the target sequence number of the target pixel point is consistent with the initial sequence number; if the target sequence number of the target pixel point is consistent with the initial sequence number, determining the target sequence number as the update sequence number of the target pixel point.

[0113] After the server determines the target sequence number of each target pixel point, it can compare the target sequence number of each target pixel point with the initial sequence number. If the two are consistent, the target sequence number is determined as the updated sequence number of the target pixel point. That is, for the target pixel point whose target sequence number is the same as the initial sequence number, the target sequence number is used as the updated sequence number of the target pixel point.

[0114] Step 5024, if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence numbers are consistent with the target sequence number of the target pixel point are used as second associated pixels, and the update sequence number of the target pixel point is determined based on the target sequence number and the initial sequence number of the second associated pixel point.

[0115] In implementation, for target pixel points whose target sequence number is inconsistent with the initial sequence number, the server can use other target pixel points whose initial sequence number is consistent with the target sequence number as second associated pixel points, and determine the update sequence number of the target pixel point based on the target sequence number and initial sequence number of the second associated pixel point.

[0116] In one implementation, the server can compare whether the target sequence number and the initial sequence number of the second associated pixel are the same. If they are the same, the target sequence number of the second associated pixel is used as the update sequence number of the target pixel. If they are not the same, other target pixels whose initial sequence numbers are consistent with the target sequence number of the second associated pixel are used as new second associated pixels, and the server returns to execute the step of comparing whether the target sequence number and the initial sequence number of the second associated pixel are the same, until a new second associated pixel whose target sequence number is the same as the initial sequence number is found, and the update sequence number of the target pixel is determined. In the next update process, the current sequence number of the target pixel can be updated to the update sequence number. Optionally, the current sequence numbers of all second associated pixels involved in the search process can be updated to the same update sequence number as the target pixel, thereby improving the update efficiency.

[0117] Step 5025, updates the current serial number of each target pixel point to the update serial number, returns to execute the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image, until the preset number of update processing times is reached, and uses the update serial number determined after the last update processing as the matching serial number, and determines the area composed of target pixel points with the same matching serial number as a connected area.

[0118] Among them, steps 5022 to 5024 can be executed in a loop, and each execution of a loop is an update process. The number of loops can be set, preferably set to 2 times. As a result, the value of the area where each moving target is located becomes the minimum serial number value or the maximum serial number value of its coverage area. If there are k targets, there will be k areas, and the value of each area is consistent, but the values ​​of different areas are different. Therefore, the area composed of target pixels with the same matching number can be determined as a connected area where a moving target is located. Optionally, the minimum circumscribed rectangle of the area where the target pixels with the same matching number are located can be used as a connected area. Specifically, the position coordinates (c, r) are placed in different arrays D respectively, which will form k arrays D, D1, D2-D k , each D contains multiple position coordinate information. For example, D1 = [(c0, r0), (c1, r1), (c2, r2)]. Calculate the minimum coordinate value (x min ,y min ) and the maximum value (x max ,y max ). The calculation formula is as follows:

[0119] x min =MIN(D[i][0])y min =MIN(D[i][1])

[0120] x max=MAX(D[i][0])y max =MAX(D[i][1])

[0121] Where i is the index of all positions in the traversal array D. The obtained (x min ,y min ) and (x max ,y max ) is the coordinates of the upper left and lower right corners of the connected domain rectangle of a moving target, which can be used to locate the position of the connected area.

[0122] In one embodiment, after the current sequence number of the target pixel point is updated to the updated sequence number in the above step 5025, the following step is also included: marking the target pixel point as a processed pixel point. Accordingly, the process of comparing whether the target sequence number and the initial sequence number of the second associated pixel point are the same specifically includes the following steps: determining whether the second associated pixel point is currently marked as a processed pixel point; if the second associated pixel point has been marked as a processed pixel point, using the updated sequence number corresponding to the second associated pixel point as the updated sequence number of the target pixel point; if the second associated pixel point has not been marked as a processed pixel point, comparing whether the target sequence number and the initial sequence number of the second associated pixel point are the same.

[0123] In one embodiment, the process of using the target sequence number of the second associated pixel point whose target sequence number is the same as the initial sequence number as the update sequence number of the target pixel point specifically includes the following steps: using the target sequence number of the second associated pixel point whose target sequence number is the same as the initial sequence number as the update sequence number, updating the current sequence numbers of the target pixel point and the second associated pixel point to the update sequence number, and marking the target pixel point and the second associated pixel point as processed pixels. Correspondingly, the process of determining whether the target sequence number and the initial sequence number of the target pixel point are consistent specifically includes the following steps: determining whether the target pixel point is currently marked as a processed pixel point, and if it is not marked as a processed pixel point, determining whether the target sequence number and the initial sequence number of the target pixel point are consistent.

[0124] In this embodiment, since the process of determining the update sequence number of each target pixel point can be processed in parallel, and in the process of determining the update sequence number of the target pixel point, the update sequence number of the second associated pixel point (also the target pixel point) involved can be determined at the same time, therefore, after determining the update sequence numbers of the target pixel point and the second associated pixel point, they can be marked as processed pixels, and then when updating the sequence number of each target pixel point, it can be determined first whether the target pixel point or the second associated pixel point has been processed, thereby improving the efficiency of connected domain analysis.

[0125] Compared with the seed filling method, the two-pass algorithm, etc., the above-mentioned connected domain analysis method has a higher degree of parallelism and can improve the efficiency of connected domain analysis and target detection.

[0126] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a device for detecting a target of a billion-pixel fused video for realizing the target detection method of the billion-pixel fused video involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the device for detecting a target of a billion-pixel fused video provided below can refer to the limitations of the target detection method of the billion-pixel fused video above, and will not be repeated here.

[0128] In one embodiment, Figure 6 As shown, a target detection device 600 for billion-pixel fusion video is provided, including: an acquisition module 601, a first detection module 602, a second detection module 603 and a third detection module 604, wherein:

[0129] The acquisition module 601 is used to acquire multiple local videos captured by the array camera.

[0130] The first detection module 602 is used to reduce the size of the current frame local video image of each channel of the local video, and input the reduced-size current frame local video image into the first target detection model for target detection to obtain a first detection result.

[0131] The second detection module 603 is used to perform motion target detection on the local video using a frame difference method, obtain candidate detection areas where each moving target is located in the local video image of the current frame, and determine a target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into the second target detection model for target detection to obtain a second detection result.

[0132] The third detection module 604 is used to obtain the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusing the multiple local videos.

[0133] In one embodiment, the first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame. The third detection module 604 is specifically used for: performing coordinate transformation on the local position information of the first target object and the second target object according to the coordinate mapping relationship between each of the local videos and the fused video, so as to obtain the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame; finding out the first target object and the second target object that meet the fused position intersection condition according to the fused position information of the first target object and the second target object, and judging whether the types of the first target object and the second target object that meet the fused position intersection condition are the same; if they are the same, merging the fused position information of the first target object and the second target object that meet the fused position intersection condition as the fused position information of the target merged object; determining the first target object and the second target object that do not meet the fused position intersection condition, the first target object and the second target object that meet the fused position intersection condition but are of different types, and the target merged object as the final target object detected in the fused video image of the current frame, so as to obtain the target detection result of the fused video image of the current frame.

[0134] In one embodiment, the third detection module 604 is specifically used to: reduce the size of the current frame fused video image of the fused video and input it into the third target detection model for target detection to obtain a third detection result; the third detection result includes the type of the detected third target object and the fusion position information of the third target object in the coordinate system of the current frame fused video image; based on the first detection result, the second detection result and the third detection result, obtain the target detection result of the current frame fused video image.

[0135] In one embodiment, the second detection module 603 is specifically used to: for each of the local videos, obtain a binary difference image corresponding to the local video image of the current frame based on the local video image of the current frame and the local video image of the historical frame of the local video; perform a connected domain analysis on the difference image to determine the connected area formed by the target pixel points, and determine the candidate detection area where each moving target is located based on each of the connected areas; the target pixel point is the pixel point corresponding to the moving target.

[0136] In one embodiment, the second detection module 603 is specifically used to: determine a historical frame local video image in the local video that is separated from the current frame local video image by a preset number of frames as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image; perform pixel-by-pixel difference calculation between each frame of the comparison frame image and the reference frame image to obtain multiple initial differential images, and binarize each of the initial differential images to obtain multiple binary initial differential images; superimpose each of the binary initial differential images, and perform morphological processing on the superimposed images to obtain a binary differential image corresponding to the current frame local video image.

[0137] In one embodiment, the second detection module 603 is specifically used to: use the initial serial number of each target pixel point in the differential image as the current serial number for the first update processing, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; for each target pixel point, search for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image as the first associated pixel points, and use the minimum or maximum current serial number of the target pixel point and the current serial number of each first associated pixel point as the target serial number of the target pixel point; determine whether the target serial number of the target pixel point is consistent with the initial serial number; if the target serial number of the target pixel point is consistent with the initial serial number, determine the target serial number as the update sequence number of the target pixel point; if the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixel points whose initial sequence number is consistent with the target sequence number of the target pixel point are used as second associated pixel points, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; the current sequence number of each target pixel point is updated to the update sequence number, and the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each target pixel point in the differential image is returned for each target pixel point, until a preset number of update processing times is reached, the update sequence number determined after the last update processing is used as the matching sequence number, and the area formed by the target pixels with the same matching sequence number is determined as a connected area.

[0138] In one embodiment, the second detection module 603 is specifically used to: compare whether the target serial number and the initial serial number of the second associated pixel point are the same; if they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, use other target pixel points whose initial serial numbers are consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to execute the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

[0139] Each module in the above-mentioned object detection device for billion-pixel fusion video can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0140] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data required or generated for executing the above-mentioned target detection method for billion-level pixel fusion video. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target detection method for billion-level pixel fusion video is implemented.

[0141] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target detection method for billion-pixel fusion video, characterized in that: The method comprises: Acquire multiple local videos captured by array cameras; Reducing the size of the local video image of the current frame of each local video, and inputting the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; Using a frame difference method to perform moving target detection on the local video, obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine a target detection area that meets a small target size condition from each of the candidate detection areas; Inputting the image of the target detection area in the local video image of the current frame into a second target detection model for target detection to obtain a second detection result; Based on the first detection result and the second detection result, a target detection result of a fused video image of a current frame of the fused video is obtained; the fused video is obtained based on the splicing and fusion of the multiple local videos.

2. The method according to claim 1, characterized in that The first detection result includes the type of the detected first target object and the local position information of the first target object in the coordinate system of the local video image of the current frame; the second detection result includes the type of the detected second target object and the local position information of the second target object in the coordinate system of the local video image of the current frame; The obtaining, based on the first detection result and the second detection result, a target detection result of the current frame fused video image of the fused video includes: According to the coordinate mapping relationship between each of the local videos and the fused video, coordinate transformation is performed on the local position information of the first target object and the second target object to obtain the fused position information of the first target object and the second target object in the coordinate system of the fused video image of the current frame; According to the fusion position information of the first target object and the second target object, the first target object and the second target object that meet the fusion position intersection condition are found, and it is determined whether the types of the first target object and the second target object that meet the fusion position intersection condition are the same; if they are the same, the fusion position information of the first target object and the second target object that meet the fusion position intersection condition are merged as the fusion position information of the target merged object; The first target object and the second target object that do not satisfy the fusion position intersection condition, the first target object and the second target object that satisfy the fusion position intersection condition but are of different types, and the target merged object are determined as the final target object detected in the current frame fused video image, and the target detection result of the current frame fused video image is obtained.

3. The method according to claim 1, characterized in that The obtaining, based on the first detection result and the second detection result, a target detection result of the current frame fused video image of the fused video includes: The size of the current frame fused video image of the fused video is reduced and then input into a third target detection model for target detection to obtain a third detection result; the third detection result includes the type of the detected third target object and the fusion position information of the third target object in the coordinate system of the current frame fused video image; Based on the first detection result, the second detection result and the third detection result, an object detection result of the current frame fused video image is obtained.

4. The method according to claim 1, characterized in that: The method of using the frame difference method to detect moving targets in the local video to obtain candidate detection areas where each moving target in the local video image of the current frame is located includes: For each of the local videos, according to the local video image of the current frame and the local video image of the historical frame, a binary difference image corresponding to the local video image of the current frame is obtained; Connected domain analysis is performed on the differential image to determine connected areas formed by target pixel points, and candidate detection areas where each moving target is located are determined based on each connected area; the target pixel point is a pixel point corresponding to the moving target.

5. The method according to claim 4, characterized in that The step of obtaining a binary difference image corresponding to the local video image of the current frame according to the local video image of the current frame and the local video image of the historical frame includes: Determine a historical frame local video image in the local video that is separated from the current frame local video image by a preset number of frames as a reference frame image, and determine each historical frame local video image between the reference frame image and the current frame local video image, and the current frame local video image as a comparison frame image; Performing pixel-by-pixel difference calculation on each comparison frame image and the reference frame image to obtain a plurality of initial differential images, and binarizing each of the initial differential images to obtain a plurality of binarized initial differential images; The binarized initial differential images are superimposed, and the superimposed images are subjected to morphological processing to obtain a binarized differential image corresponding to the local video image of the current frame.

6. The method according to claim 4, characterized in that The performing connected domain analysis on the differential image to determine the connected area formed by the target pixel points includes: Using the initial serial number of each target pixel point in the differential image as the current serial number of the first update process, the initial serial number is an ascending or descending serial number pre-assigned to each target pixel point in the differential image; For each of the target pixel points, in the differential image, along the extension direction of the neighboring pixel points of each of the target pixel points, other target pixel points connected in each extension direction are searched as first associated pixel points, and the minimum or maximum current sequence number of the target pixel point and the current sequence numbers of each of the first associated pixel points is used as the target sequence number of the target pixel point; Determine whether the target sequence number and the initial sequence number of the target pixel are consistent; if the target sequence number and the initial sequence number of the target pixel are consistent, determine the target sequence number as the update sequence number of the target pixel; If the target sequence number and the initial sequence number of the target pixel point are inconsistent, other target pixels whose initial sequence numbers are consistent with the target sequence number of the target pixel point are used as second associated pixels, and the update sequence number of the target pixel point is determined according to the target sequence number and the initial sequence number of the second associated pixel point; The current serial number of each of the target pixel points is updated to the update serial number, and the step of searching for other target pixel points connected in each extension direction along the extension direction of the adjacent pixel points of each of the target pixel points in the differential image is returned to be executed for each of the target pixel points until a preset number of update processing times is reached, and the update serial number determined after the last update processing is used as the matching serial number, and the area formed by the target pixel points with the same matching serial number is determined as a connected area.

7. The method according to claim 6, characterized in that The step of determining the update sequence number of the target pixel point according to the target sequence number and the initial sequence number of the second associated pixel point comprises: Compare whether the target serial number and the initial serial number of the second associated pixel point are the same. If they are the same, determine the target serial number of the second associated pixel point as the updated serial number of the target pixel point; if they are not the same, take other target pixel points whose initial serial numbers are consistent with the target serial number of the second associated pixel point as new second associated pixel points, and return to execute the step of comparing whether the target serial number and the initial serial number of the second associated pixel point are the same.

8. An object detection device for billion-pixel fusion video, characterized in that: The device comprises: An acquisition module is used to acquire multiple local videos captured by array cameras; A first detection module is used to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame into a first target detection model for target detection to obtain a first detection result; The second detection module is used to perform moving target detection on the local video by using a frame difference method, obtain candidate detection areas where each moving target is located in the local video image of the current frame, and determine a target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame into a second target detection model for target detection, and obtain a second detection result; The third detection module is used to obtain the target detection result of the current frame fused video image of the fused video based on the first detection result and the second detection result; the fused video is obtained by splicing and fusion of the multiple local videos.

9. A billion-pixel computational imaging system, characterized in that: The billion-pixel computational imaging system comprises an array camera, a server and a display terminal, wherein the array camera comprises a plurality of sub-cameras, wherein: The array camera is used to shoot a video of a target scene through multiple sub-cameras, and send the shot multiple local videos to a server; The server is used for splicing and fusing the acquired local videos to obtain a fused video; The server is also used to reduce the size of the local video image of the current frame of each local video, and input the reduced-size local video image of the current frame to the first target detection model for target detection to obtain a first detection result; use the frame difference method to perform motion target detection on the local video to obtain candidate detection areas where each moving target in the local video image of the current frame is located, and determine the target detection area that meets the small target size condition from each candidate detection area; input the image of the target detection area in the local video image of the current frame to the second target detection model for target detection to obtain a second detection result; based on the first detection result and the second detection result, obtain the target detection result of the fused video image of the current frame of the fused video; The server is further configured to send the fused video and the target detection result to the display terminal.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images