Large-view-field target real-time detection tracking system based on array camera
Through the multi-channel video acquisition, splicing and object detection and tracking module based on array cameras, the problem of limited field of view and synchronization of traditional systems is solved, and multi-objective real-time detection and tracking under large field of view is realized, meeting the needs of large field of view monitoring.
Patent Information
- Application Number
- CN202411277178.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional single-camera-based object detection and tracking system has limited field of view and is difficult to cover large-scale scenarios. The array camera system has technical difficulties such as synchronization, calibration and data fusion in real-time detection and tracking.
A large field of view target real-time detection and tracking system based on array cameras is designed, including multiple video acquisition modules, splicing modules and target detection and tracking modules. Through the coordinated work of multiple cameras, video streams are collected in real time and utilizing homography. The transformation matrix performs image stitching to form a large field of view video stream, and achieves real-time detection and tracking of the target through shared memory transmission and YOLOv5 detector.
It realizes real-time detection and tracking of multi-objectives under large field of view, overcomes the problem of small detection and tracking range of traditional systems, meets the needs of large field of view monitoring, and provides adaptive splicing and real-time detection and tracking effects.
Smart Images

Figure CN119094707A_ABST
Abstract
Description
1. Technical Field
[0001] The present invention relates to the field of computer vision, and more specifically to a large-field-of-view target real-time detection and tracking system based on an array camera. 2. Background Technology
[0002] Target detection and tracking is the core of modern surveillance systems. It simulates the human visual system to identify, locate and track targets in images or videos, playing an important role in many fields, including but not limited to security monitoring, traffic management, military reconnaissance and other fields. Applying target detection and tracking in the above fields can analyze dynamic scenes in real time, respond to emergencies quickly, reduce the need for manual monitoring, reduce labor costs and improve efficiency. In addition, it can track multiple targets to provide more comprehensive monitoring coverage.
[0003] Although object detection and tracking technology has significant advantages, it still faces some challenges in practical applications: the traditional single-camera-based object detection and tracking system has a limited field of view and is difficult to cover a large range of scenes. While pursuing high resolution, the system often sacrifices the breadth of the field of view, resulting in limited application scenarios.
[0004] In order to overcome the above limitations, researchers have developed array camera systems that achieve large field of view and high-resolution video acquisition through the collaborative work of multiple cameras. However, these systems also have some problems, such as the integration of array camera systems, which has technical difficulties such as multi-camera synchronization, calibration, and data fusion, making it difficult to perform real-time detection and tracking. III. Summary of the invention
[0005] To solve the above problems, the present invention proposes a large-field-of-view target real-time detection and tracking system based on an array camera. The system includes a multi-channel video acquisition module, a splicing module, a transmission module and a target detection and tracking module.
[0006] The multi-channel video acquisition module includes a plurality of cameras and a fixing bracket, and is used for real-time synchronous acquisition of multi-channel video streams. The viewing areas of adjacent cameras partially overlap.
[0007] The splicing module is used to perform real-time splicing of the collected multi-channel video streams to obtain a large-field-of-view video stream. First, the overlapping area between cameras is extracted, feature point detection and matching are performed, and the camera image is perspective transformed using the homography transformation matrix, and then the multi-camera collection images are spliced to form a large-field-of-view video stream.
[0008] The transmission module is used to transmit the spliced large-field-of-view video stream frame by frame to the transfer area cache built by the shared memory.
[0009] The target detection and tracking module is used to detect and identify specific categories of targets in a large field of view video screen, and track the position and motion information of the targets in the scene in real time.
[0010] Furthermore, in the multi-channel video acquisition module, the cameras can be arranged at equal intervals along one or two dimensions, wherein each camera acquires a video of a single perspective of a scene from a specific orientation. The camera can perform pitch, yaw and roll rotation operations within a certain range.
[0011] Furthermore, the transfer module and the transfer area allow multiple processes or modules to access the same memory area.
[0012] Furthermore, the target detection and tracking module detects the target in each frame of the spliced large-field-of-view image in real time, identifies the target and marks the position of the target in the image with a square frame, thereby achieving a real-time detection and tracking effect.
[0013] The present invention proposes a large-field-of-view target real-time detection and tracking system based on an array camera, and the workflow comprises the following steps: collecting a plurality of camera images to obtain a plurality of video frames at the same time; finding feature points of each video frame and matching key feature points; calculating the optimal homography transformation matrix between a plurality of feature point pairs and estimating the homography matrix corresponding to each camera; performing perspective transformation and exposure compensation on the video frames of each camera and splicing and synthesizing large-field-of-view video frames; constructing a shared memory structure, transmitting the spliced large-field-of-view video stream frame by frame and storing it in a transfer area, wherein the target detection and tracking module periodically extracts each video frame data from the transfer area, performs detection, recognition and tracking, and completes the real-time detection and tracking of multiple targets under a large field of view. IV. Description of the drawings
[0014] The above aspects and advantages of the present invention will be further clarified and easily understood from the following detailed description in conjunction with the accompanying drawings and embodiments, in which:
[0015] Figure 1 Schematic diagram of system composition according to an embodiment of the present application.
[0016] Figure 2 Schematic diagram of the structure of a dual-channel video acquisition module according to an embodiment of the present application.
[0017] Figure 3 Schematic diagram of the acquisition and target detection and tracking range according to an embodiment of the present application.
[0018] Figure 4 It is a schematic diagram showing a transfer area according to an embodiment of the present application.
[0019] Figure 5 It is a schematic diagram demonstrating the system operation according to an embodiment of the present application.
[0020] The diagram numbers in the above drawings are:
[0021] 101 dual-channel video acquisition module, 102 splicing module, 103 transmission module, 104 target detection and tracking module, 201 left camera, 202 right camera, 203 processing chip, 204 housing, 205 rotating shaft, 206 fixed bracket, 302 pedestrian target I, 303 pedestrian target II, 304 pedestrian target III, 401 transfer area, 402 first frame of transfer area queue, 501 host computer, 502 display. V. Specific implementation methods
[0022] In order to facilitate the understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings. Optional embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive.
[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the invention pertains. The terms used herein in the specification of the invention are for the purpose of describing specific embodiments only and are not intended to limit the present application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.
[0024] A large-field-of-view target real-time detection and tracking system based on an array camera proposed in the present application will be described in detail below in conjunction with the embodiments and drawings disclosed in the present application.
[0025] Figure 1 The schematic diagram of the system composition according to an embodiment of the present application is shown. The system is composed of a dual-channel video acquisition module 101, a splicing module 102, a transmission module 103 and a target detection and tracking module 104.
[0026] The dual-channel video acquisition module 101 includes two cameras and a fixed bracket, which is used to synchronously acquire dual-channel video streams in real time. Figure 2As shown, the left camera 201, the right camera 202, the processing chip 203, the housing 204, the rotating shaft 205 and the fixing bracket 206 together constitute the dual-channel video acquisition module 101. The fields of view of adjacent cameras partially overlap. The two cameras are arranged at equal intervals in the horizontal direction, and respectively collect video data of the scene from specific left and right directions. The camera can perform pitch, yaw and roll rotation operations within a certain range. The pitch angle, yaw angle and roll angle of the two cameras are adjusted so that the fields of view of the two cameras face outward and the two fields of view partially overlap. The resolution of the camera is 1280×720 pixels, and the acquisition frame rate is 30 frames per second.
[0027] In this embodiment, the number of additional video frames captured by the left camera 201 is determined by measuring the initialization time and multiplying it by the acquisition frame rate. The same video frame of the left camera 201 and the right camera 202 is read cyclically to synchronize the images captured by the two cameras, and a suitable stitching area is found within the field of view of the two cameras. The captured dual-channel videos are synchronously transmitted frame by frame to the stitching module 102 in real time.
[0028] Figure 3 A schematic diagram of the acquisition and target detection tracking range according to an embodiment of the present application is shown. The splicing module 102 is used to perform real-time splicing of the two-way video streams collected by the two-way video acquisition module 101 to obtain a large field of view video stream. Specifically, the two-way video frames input by the left camera 201 and the right camera 202 are cropped respectively, the image size is adjusted, the overlapping area between the left camera field of view 305 and the left camera field of view 307 is extracted, the SURF detector is called to find feature points, the key feature points of the overlapping image area 306 are calculated for matching, the matching results are filtered, the optimal matching pair is found, and the picture calibration and matching are realized; the optimal homography transformation matrix between multiple two-dimensional feature point pairs is calculated, the homography matrix corresponding to the left camera 201 and the right camera 202 is estimated, and the homography matrix is robustly estimated using a random sampling consistency algorithm to optimize the homography matrix; the video frames of the left camera 201 and the right camera 202 are perspective transformed using the homography transformation matrix, the transformed video frames are exposed compensated, the gap effect is eliminated, and the large field of view video frames are spliced and synthesized, and transmitted to the transmission module. The stitched large field of view includes pedestrian target I 302 , pedestrian target II 303 , and pedestrian target III 304 .
[0029] The transmission module 103 is used to transmit the large field of view video stream generated by the splicing module 102 frame by frame to the transfer area cache built by the shared memory, and the target detection and tracking module 104 extracts the large field of view video stream frame by frame. The transfer area allows multiple processes or modules to access the same memory area.
[0030] Figure 4The transfer area demonstration schematic diagram according to an embodiment of the present application is shown. The transfer area receives the large field of view video frames processed by the splicing module 102 and stores them in the transfer area 401 in the form of a queue. Each time, the first frame 402 of the transfer area queue is taken to the target detection and tracking module 104 for target detection and tracking.
[0031] The target detection and tracking module 104 is equipped with a pre-trained data set of YOLOv5, calls YOLOv5 to detect and identify all pedestrian targets in a large field of view video screen in real time, and tracks the position and motion information of the pedestrian targets in the scene in real time to obtain a complete spliced video screen with detection and tracking results with a detection frame.
[0032] Figure 5 A schematic diagram of a system working demonstration according to an embodiment of the present application is shown. The workflow of the large field of view target real-time detection and tracking system based on an array camera includes the following steps: the dual-channel video acquisition module 101 acquires the images of the left camera 201 and the right camera 202 through clock synchronization to obtain several video frames at the same time; each video frame is cropped, the SURF detector is called to find feature points, key feature points are calculated for matching, the matching results are filtered, and the optimal matching pair is found; the optimal homography transformation matrix between multiple two-dimensional feature point pairs is calculated, the homography matrix corresponding to the left camera 201 and the right camera 202 is estimated, and the homography transformation matrix is robustly estimated using a random sampling consistency algorithm to improve the homography. The accuracy and stability of the homography transformation matrix are improved; the video frames of the left camera 201 and the right camera 202 are respectively transformed in perspective using the homography transformation matrix, exposure compensation is performed on the transformed video frames, the gap effect is eliminated, and the large field of view video frames are spliced together; a shared memory structure is constructed on the host computer 501, and the spliced large field of view video stream is transmitted frame by frame and stored in the transfer area 401, the target detection and tracking module 104 continuously extracts the first frame 402 data of the queue from the transfer area, uses the YOLOv5 detector for detection, recognition and tracking, completes the detection and tracking of the pedestrian target 302 in real time with a large field of view, and outputs the video signal to the display 502.
[0033] The advantages of the large-field-of-view real-time detection and tracking system for targets based on an array camera proposed in the present invention are that it can adaptively complete large-field-of-view splicing and real-time detection and tracking of specific categories of targets in a large-field-of-view image, overcome the problem of a small detection and tracking range of a traditional target detection and tracking system, achieve real-time large-field-of-view target detection and tracking effects, and meet the needs of large-field-of-view monitoring.
Claims
1. A large-field-of-view target real-time detection and tracking system based on an array camera, characterized in that: The system includes a multi-channel video acquisition module, a splicing module, a transmission module and a target detection and tracking module; The multi-channel video acquisition module includes several cameras and a fixed bracket, which is used to synchronously acquire multiple video streams in real time, and the viewing areas of adjacent cameras partially overlap; The splicing module is used to perform real-time splicing of the collected multi-channel video streams to obtain a large-field-of-view video stream; first, the overlapping area between the cameras is extracted, feature point detection and matching are performed, the camera image is perspectively transformed using the homography transformation matrix, and then the multi-camera collection images are spliced to form a large-field-of-view video stream; The transmission module is used to transmit the spliced large-field-of-view video stream frame by frame to the transfer area cache built by the shared memory; The target detection and tracking module is used to detect and identify specific categories of targets in a large field of view video screen, and track the position and motion information of the targets in the scene in real time.
2. According to claim 1, a large field of view target real-time detection and tracking system based on an array camera is characterized in that: In the multi-channel video acquisition module, the cameras can be arranged at equal intervals along one or two dimensions, wherein each camera acquires video of a single perspective of a scene from a specific orientation, and the cameras can perform pitch, yaw and roll rotation operations within a certain range.
3. According to claim 1, a large field of view target real-time detection and tracking system based on an array camera is characterized in that: The transmission module and the transfer area allow multiple processes or modules to access the same memory area.
4. The large-field-of-view target real-time detection and tracking system based on an array camera according to claim 1, characterized in that: The target detection and tracking module detects the target in each frame of the spliced large-field-of-view image in real time, identifies the target and marks the position of the target in the image with a square frame, thereby achieving a real-time detection and tracking effect.
5. The large-field-of-view target real-time detection and tracking system based on an array camera according to claim 1, characterized in that: The workflow of the system includes the following steps: Collect several camera images to obtain several video frames at the same time; Find the feature points of each video frame and match the key feature points; Calculate the optimal homography transformation matrix between multiple feature point pairs and estimate the homography matrix corresponding to each camera; Perform perspective transformation and exposure compensation on the video frames of each camera, and stitch them together to form a large-field-of-view video frame; A shared memory structure is constructed to transmit and store the spliced large-field-of-view video stream frame by frame to the transfer area. The target detection and tracking module periodically extracts each video frame data from the transfer area for detection, recognition and tracking, thereby completing real-time detection and tracking of multiple targets under a large field of view.
Citation Information
Patent Citations
Cross-area moving object detection method and apparatus based on video splicing
CN107948586A
Multi-camera target detection tracking method and device
CN115457084A
Video stitching-based multi-camera collaborative swimming speed measurement method and system
CN116309685A
Interaction with three-dimensional video
US20160086379A1
