Multi-camera fusion perception method, system, electronic device and storage medium

Abstract

The present invention provides a multi-camera fusion perception method, system, electronic device and storage medium, which relates to the field of intelligent transportation technology. The method comprises: obtaining current video frames output by multiple cameras deployed at the same intersection; performing 2D target recognition and tracking and target motion state estimation on the current video frames output by each camera, and obtaining the original perception result corresponding to each camera; wherein the original perception result includes a timestamp, a target identifier and motion state information; based on the motion state similarity between the targets, the original perception results corresponding to each camera are subjected to fusion tracking of the identified target, and the current fusion perception result is obtained. In this way, target fusion between multiple cameras at the intersection is realized, a unified target perception result can be obtained, and the target motion state estimation accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a multi-camera fusion perception method, system, electronic equipment and storage medium. Background Art

[0002] Cameras are the most widely used sensor devices in traffic systems. With the development of applications in the fields of smart transportation and vehicle-road collaboration, more requirements are put forward for the perception capabilities of cameras. For example, in addition to the basic requirements of vehicle target detection and classification, cameras must also be able to track vehicle trajectories and measure information such as position, speed, acceleration, and heading angle. By sensing the number and speed of vehicles at intersections, traffic management departments can be helped to promptly detect and deal with congestion, traffic accidents, and other problems, ensuring road traffic safety.

[0003] Currently, camera perception at intersections faces the following problems: 1. Insufficient accuracy in estimating the target’s motion state; 2. Limited field of view of a single camera; 3. Inconsistent targets between multiple cameras. Summary of the invention

[0004] The object of the present invention is to provide a multi-camera fusion perception method, system, electronic device and storage medium to solve at least one of the above problems.

[0005] In a first aspect, an embodiment of the present invention provides a multi-camera fusion perception method, including:

[0006] Get the current video frames output by multiple cameras deployed at the same intersection;

[0007] Perform 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera to obtain the original perception result corresponding to each camera; the original perception result includes timestamp, target identification and motion state information;

[0008] Based on the similarity of the motion states between the targets, the original perception results corresponding to each camera are fused and tracked to identify the target, and the current fused perception result is obtained.

[0009] Furthermore, 2D target recognition and tracking and target motion state estimation are performed on the current video frame output by each camera to obtain the original perception results corresponding to each camera, including:

[0010] Through the preset 2D detection algorithm and 2D tracking algorithm, the current video frame output by each camera is subjected to 2D target recognition and tracking, and the 2D structured information of the recognized target corresponding to each camera is obtained; wherein the 2D structured information of the recognized target includes a timestamp, a target identifier and a 2D detection frame;

[0011] According to the 2D structural information of the recognition target corresponding to each camera and the preset installation viewing angle information, determine the 2D reference point of the recognition target in the corresponding current video frame; wherein, as the distance between the recognition target and the camera decreases, the 2D reference point of the recognition target gradually changes from the center point of the bottom edge of the 2D detection frame close to the recognition target to the center point of the 2D detection frame close to the recognition target;

[0012] Determine the original 3D position information of the identified target corresponding to each camera according to the preset transformation matrix corresponding to each camera and the 2D reference point of the identified target in the corresponding current video frame; wherein the original 3D position information of the identified target is used to indicate the actual position of the identified target;

[0013] Through the pre-established uniform acceleration motion model and Kalman filter, the original 3D position information of the identified target corresponding to each camera is filtered, and the speed, acceleration and heading angle are obtained to obtain the original perception result corresponding to each camera.

[0014] Further, according to the 2D structured information of the recognition target corresponding to each camera and the preset installation viewing angle information, a 2D reference point of the recognition target in the corresponding current video frame is determined, including:

[0015] According to the 2D structured information of the recognition target corresponding to each camera, the center point coordinates and height of the 2D detection frame of the recognition target are determined;

[0016] The 2D reference point of the identified target is calculated by the following formula:

[0017] ;

[0018] in, u ref_2D Indicates the horizontal coordinate of the 2D reference point, v ref_2D Indicates the ordinate of the 2D reference point, u c Represents the horizontal coordinate of the center point of the 2D detection box, v c Indicates the ordinate of the center point of the 2D detection box, v v Indicates the vertical coordinate of the preset vanishing point corresponding to the camera, h Indicates the height of the 2D detection box, H Indicates the image height of the current video frame. δ Indicates the zoom scale corresponding to the preset installation viewing angle information of the camera.

[0019] Furthermore, based on the similarity of the motion states between the targets, the original perception results corresponding to each camera are fused and tracked to identify the target, and the current fused perception results are obtained, including:

[0020] Preprocessing the original perception results corresponding to each camera to obtain multiple preprocessed perception results;

[0021] Based on the similarity of the motion states between the targets and the stored previous fusion perception results, target matching and target fusion are performed on each pre-processed perception result to obtain the current fusion result;

[0022] Perform target management on the current fusion result to obtain a fusion perception result; wherein the target management includes creating new targets and / or deleting disappeared targets.

[0023] Furthermore, the original perception results corresponding to each camera are preprocessed to obtain multiple preprocessed perception results, including:

[0024] According to the timestamp in the original perception result corresponding to each camera, the candidate perception result corresponding to the synchronous video frame is obtained;

[0025] The candidate perception results are normalized in coordinate system and the recognition targets in the preset region of interest are screened to obtain the preprocessed perception results.

[0026] Furthermore, based on the similarity of the motion states between the targets and the stored previous fusion perception results, target matching and target fusion are performed on each pre-processed perception result to obtain the current fusion result, including:

[0027] Predict the current fusion perception result according to the previous fusion perception result to obtain the current predicted perception result;

[0028] Performing target association of target identification on the identified target in each preprocessed perception result and the fused target in the current predicted perception result to obtain a preliminary matching result;

[0029] Determine whether there are unassociated identification targets in the preliminary matching results;

[0030] If there are unassociated recognition targets, the unassociated recognition targets in the preliminary matching results are associated with the fusion targets in the current predicted perception results by using the motion state information to obtain the final matching results.

[0031] Through the pre-established uniform acceleration motion model and Kalman filter, the successfully associated identification targets in the final matching results are fused to obtain the current fusion result.

[0032] Furthermore, the current fusion result is managed by target to obtain the fusion perception result, including:

[0033] When there is a first target that meets the preset new target condition in the current fusion result, a fusion target corresponding to the first target is created; wherein the new target condition includes that it is not successfully associated with the fusion target corresponding to the previous fusion perception result;

[0034] When there is a second target that meets the preset target disappearance condition in the current fusion result, the second target is deleted; wherein the target disappearance condition includes that the number of continuous tracking successes corresponding to the newly created fusion target does not reach the first preset number, or the number of continuous tracking failures corresponding to the fusion target reaches the second preset number.

[0035] In a second aspect, an embodiment of the present invention further provides a multi-camera fusion perception system, including:

[0036] An acquisition module is used to acquire current video frames output by multiple cameras deployed at the same intersection;

[0037] The perception module is used to perform 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera, and obtain the original perception result corresponding to each camera; wherein the original perception result includes timestamp, target identification and motion state information;

[0038] The fusion module is used to perform fusion tracking of the identified targets on the original perception results corresponding to each camera based on the similarity of the motion states between the targets, and obtain the current fusion perception results.

[0039] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the multi-camera fusion perception method of the first aspect is implemented.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the multi-camera fusion perception method of the first aspect is executed.

[0041] The multi-camera fusion perception method, system, electronic device and storage medium provided by the embodiment of the present invention can obtain the current video frames output by multiple cameras deployed at the same intersection; perform 2D target recognition and tracking and target motion state estimation on the current video frames output by each camera to obtain the original perception results corresponding to each camera; wherein the original perception results include timestamp, target identification and motion state information; based on the motion state similarity between the targets, the original perception results corresponding to each camera are fused and tracked to identify the target to obtain the current fused perception results. In this way, target fusion between multiple cameras at the intersection is achieved, a unified target perception result can be obtained, and the target motion state estimation accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a multi-camera fusion perception method provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the implementation principle of a multi-camera fusion perception method provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a 2D reference point calculation principle provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the implementation principle of the fusion tracking process of identifying a target in a multi-camera fusion perception method provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a multi-camera fusion perception system provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] At present, camera perception at intersections faces the following problems:

[0051] 1. Insufficient accuracy in target motion state estimation: After the camera acquires the video, it can use the target detection algorithm to realize 2D target recognition. However, to estimate the target's position, speed, heading angle and other motion state information, it needs to be converted into 3D space (i.e., actual space). This involves the conversion calculation from 2D pixels to 3D space points. Existing solutions usually do not consider the impact of different target positions and camera viewing angles on the position estimation accuracy, as well as the drift of the 3D position caused by the jitter of the 2D detection frame.

[0052] 2. Limited field of view: A single camera can only provide a single-directional perspective. For intersection application scenarios, it is usually necessary to reasonably deploy multiple cameras to achieve full-directional perception coverage.

[0053] 3. Inconsistent targets between multiple cameras: Since each camera detects and tracks the target independently, it is impossible to keep tracking the target when it moves from the perception range of one camera to the perception range of another camera. This causes the motion trajectory of the same target to be fragmented, affecting subsequent applications.

[0054] Based on this, the embodiments of the present invention provide a multi-camera fusion perception method, system, electronic device and storage medium, which can achieve continuous target tracking and improve the estimation accuracy of the target motion state through target fusion between multiple cameras at the intersection.

[0055] To facilitate understanding of this embodiment, a multi-camera fusion perception method disclosed in an embodiment of the present invention is first introduced in detail.

[0056] The embodiment of the present invention provides a multi-camera fusion perception method, which can be executed by an electronic device with image processing capability, and the electronic device is connected to multiple cameras deployed at the same intersection. Figure 1 The flowchart of a multi-camera fusion perception method shown in FIG. 1 mainly includes the following steps S110 to S130:

[0057] Step S110, obtaining current video frames output by multiple cameras deployed at the same intersection.

[0058] In this embodiment, multiple cameras can be deployed at different locations at the same intersection, and the installation angles of each camera are usually different. In order to fully cover the traffic intersection area, multiple cameras can be deployed in different directions of the intersection to monitor the intersection, and the cameras have overlapping coverage areas to better support subsequent target fusion between multiple cameras. It should be noted that the number of cameras deployed at the same intersection can be set according to actual needs, and there is no limitation here. For example, two, three or four cameras can be deployed at the same intersection.

[0059] The electronic device executing the method can pull RTSP (Real Time Streaming Protocol) video streams from each camera, obtain video frames, and record the current timestamp of each video frame for time synchronization of multi-camera fusion. The current video frame is the video frame to be processed, and the electronic device can process the current video frame output by each camera in real time.

[0060] To ensure that multiple cameras have consistent time, you can use NTP (Network Time Protocol) to calibrate the cameras. NTP synchronizes the clocks of each camera with a reference time source (usually an atomic clock or GPS (Global Positioning System) time source) through the Internet or a local area network to ensure that each camera has a unified time standard.

[0061] Step S120, performing 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera, and obtaining the original perception result corresponding to each camera; wherein the original perception result includes a timestamp, a target identifier and motion state information.

[0062] The above-mentioned motion state information may include information such as 3D position, speed, acceleration and heading angle, among which 3D position refers to the position of the identified target in the actual 3D space, that is, the actual position. The original perception result may also include information such as target type, and the target type may be one of multiple pre-set types, such as people, motor vehicles or non-motor vehicles.

[0063] 2D detection algorithm and 2D tracking algorithm can be used for 2D target recognition and tracking. The target obtained by 2D target recognition and tracking is called the recognized target. The current video frame is processed by the 2D detection algorithm and the 2D tracking algorithm to obtain the 2D structured information of the recognized target, including timestamp, target ID (i.e. target identification), target type and 2D detection frame (used to characterize the location of the region where the recognized target is located). Based on the structured information of the recognized target output after 2D target recognition and tracking, the target motion state can be calculated (i.e., target motion state estimation). The target motion state estimation can be divided into two sub-steps: position estimation and Kalman filtering.

[0064] In some possible embodiments, the above step S120 may include the following sub-steps S121 to S124:

[0065] Sub-step S121, through the preset 2D detection algorithm and 2D tracking algorithm, 2D target recognition and tracking are performed on the current video frame output by each camera to obtain the 2D structured information of the recognized target corresponding to each camera; wherein the 2D structured information of the recognized target includes a timestamp, a target identifier and a 2D detection frame.

[0066] The above 2D detection algorithm can be selected according to actual needs. Optionally, the 2D detection algorithm can include two types: sliding window-based methods and deep learning-based methods, wherein the sliding window-based methods may include Viola-Jones and HOG (Histogram of Oriented Gradients) + SVM (Support Vector Machine), etc., and the deep learning-based methods may include R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN, Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector) and RetinaNet, etc.

[0067] In a possible implementation, in order to detect targets in multiple camera video streams in real time, the Yolov5s model is used in an embodiment of the present invention to detect targets such as pedestrians, motor vehicles and non-motor vehicles. The YOLOv5s model is suitable for running on resource-constrained devices. The target recognition information output by the 2D detection algorithm may include the target 2D box, target type and confidence score. The confidence score can be an object confidence score, a category confidence score or a comprehensive confidence score obtained based on the object confidence score and the category confidence score; wherein the object confidence score measures the probability that the model believes that there is a target in a certain bounding box; the category confidence score measures the probability that the model believes that the target in a certain bounding box belongs to a specific category.

[0068] The goal of the above 2D tracking algorithm is to continuously estimate the position of the target in the video sequence, and the 2D tracking algorithm can be selected according to actual needs. Optionally, the 2D tracking algorithm can include three categories: template matching-based methods, feature-based methods, and deep learning-based methods. Among them, the template matching-based methods may include Mean Shift and CamShift, etc., the feature-based methods may include KCF (Kernelized Correlation Filters) and TLD (Tracking-Learning-Detection), etc., and the deep learning-based methods may include Siamese Networks, MDNet (Multi-Domain Network), SiamRPN and DaSiamRPN, etc.

[0069] In a possible implementation, the Yolov5s model obtains the target 2D frame and uses the image multi-target tracking algorithm to track the target in the image. In the embodiment of the present invention, the ByteTrack algorithm is used as the 2D multi-target tracking algorithm, and the ByteTrack algorithm can achieve real-time operation. The structured information of the identified target finally output by the 2D tracking algorithm may include: timestamp, target ID, target type, and 2D detection frame, etc.

[0070] Sub-step S122, determines the 2D reference point of the identified target in the corresponding current video frame according to the 2D structured information of the identified target corresponding to each camera and the preset installation viewing angle information; wherein the 2D reference point of the identified target gradually changes from the center point of the bottom edge of the 2D detection frame close to the identified target to the center point of the 2D detection frame close to the identified target as the distance between the identified target and the camera decreases.

[0071] When performing position estimation, you need to first select the 2D reference point of each identified target. The 2D reference point is used to indicate the position of the identified target in the corresponding current video frame. Considering that the bottom center of the 3D box of the identified target is the position of the identified target in the corresponding current video frame, that is, the ideal position of the 2D reference point, the closer the distance between the identified target and the camera, the closer the bottom center of the 3D box is to the center point of the 2D detection box, and the farther the distance is, the closer the bottom center of the 3D box is to the bottom center point of the 2D detection box. Based on this, the width direction and height direction of the 2D detection box can be used as the horizontal axis and the vertical axis respectively, and the upper left corner of the current video frame is the origin of the coordinate system to construct an image coordinate system, and determine the 2D reference point of the identified target in the following way:

[0072] According to the 2D structured information of the identified target corresponding to each camera, the center point coordinates and height of the 2D detection frame of the identified target are determined; the 2D reference point of the identified target is calculated by the following formula:

[0073] ;

[0074] in, u ref_2D Indicates the horizontal coordinate of the 2D reference point, v ref_2D Indicates the ordinate of the 2D reference point, u c Represents the horizontal coordinate of the center point of the 2D detection box, v c Indicates the ordinate of the center point of the 2D detection box, v v Indicates the ordinate of the preset vanishing point corresponding to the camera. The preset vanishing point is the intersection point obtained by extending the parallel lines of lane lines, curbs, etc. in the video frame image based on the principle of projective geometry. h Indicates the height of the 2D detection box,H Indicates the image height of the current video frame. δ Indicates the zoom scale corresponding to the preset installation viewing angle information of the camera. δ ∈(0,1).

[0075] The 2D reference point obtained in this way is closer to the center point of the 2D detection frame when the distance between the target and the camera is close, and closer to the center point of the bottom edge of the 2D detection frame when the distance between the target and the camera is far.

[0076] Sub-step S123, determines the original 3D position information of the identified target corresponding to each camera according to the preset transformation matrix corresponding to each camera and the 2D reference point of the identified target in the corresponding current video frame; wherein the original 3D position information of the identified target is used to indicate the actual position of the identified target.

[0077] The preset transformation matrix is ​​a homography matrix, which is used to describe the projection transformation relationship between two planes. The preset transformation matrix corresponding to each camera is obtained by calibrating the parameters of the camera. For example, the preset transformation matrix corresponding to the camera is calculated based on the pixel coordinates of the target in the video frame image taken by a camera and the spatial point coordinates in the actual space. The spatial point coordinates can use the longitude and latitude obtained from the GPS device or the high-precision map. In order to facilitate subsequent calculations, the longitude and latitude are uniformly converted to the coordinates of the UTM (Universal Transverse Mercator Grid System) coordinate system; among them, UTM is a map projection method, which is widely used in reference grids of topographic maps and satellite images. The deformation of the angle is minimized during the projection process. It is a plane rectangular coordinate system.

[0078] After getting the 2D reference point ( u,v ), the original 3D position can be calculated according to the following formula ( x,y,z ):

[0079] ;

[0080] in, H is the preset transformation matrix; considering the target is on the ground, z=0.

[0081] In sub-step S124, the original 3D position information of the identified target corresponding to each camera is filtered through the pre-established uniform acceleration motion model and Kalman filter, and the velocity, acceleration and heading angle are obtained to obtain the original perception result corresponding to each camera.

[0082] In order to reduce the 3D position drift caused by the jitter of the 2D detection frame, the original 3D position information of the identified target can be Kalman filtered. The state vector and target position of the identified target in the previous video frame (i.e., the position obtained after Kalman filtering) can be used to first obtain the measurement vector corresponding to the current video frame based on the target position of the identified target in the previous video frame and the original 3D position of the identified target in the current video frame; the state vector of the identified target in the previous video frame is brought into the uniform acceleration motion model to obtain the predicted state vector corresponding to the current video frame, and the state vector corresponding to the current video frame is obtained based on the predicted state vector and the measurement vector corresponding to the current video frame, that is, the position, velocity, acceleration and other information after Kalman filtering are obtained.

[0083] For ease of understanding, the Kalman filter process is described in detail below. Kalman filtering includes two steps: target state prediction and update. Specifically, the Kalman filter state vector is defined as ,in, , is the target location, , is the target speed, , is the target acceleration; the measurement vector is ,in, , is the initial position of the identified target in the current video frame (i.e. the original 3D position information), , The initial velocity is calculated from the initial position of the identified target in the current video frame and the target position of the identified target in the previous video frame (ie, the position obtained after Kalman filtering).

[0084] The uniform acceleration motion model is represented by a state transfer matrix, which can be:

[0085] ;

[0086] in, t It is the interval between the current video frame and the previous video frame.

[0087] In this embodiment, the measurement matrix of the Kalman filter is defined as:

[0088] ;

[0089] The Kalman filter process can be as follows:

[0090] ① Kalman filter prediction step:

[0091] Predict the current frame based on the state of the previous frame (i.e., based on the state vector of the previous video frame The predicted state vector corresponding to the current video frame is ), the state prediction process is:

[0092] ;

[0093] in, P It is the Kalman filter state covariance matrix, and the initial value needs to be manually set in the first frame; Q is the process noise covariance matrix, and the initial value needs to be set manually.

[0094] ② Kalman filter update step:

[0095] First, the Kalman gain is calculated, and then the update state is calculated based on the predicted prior state (i.e., the predicted state vector corresponding to the current video frame) and the measured value of the current video frame (i.e., the measurement vector). The Kalman gain K for:

[0096] ;

[0097] in, R is the measurement covariance matrix, and the initial value needs to be set manually. The Kalman filter update process is:

[0098] ;

[0099] This is the result after Kalman filtering.

[0100] The position, velocity, acceleration and other information after Kalman filtering are more consistent with the actual state of the target, reducing the problem of position jitter and jump. The target heading angle can be calculated using the average value of the velocity obtained by Kalman filtering, and the average velocity of the nearest preset number of relevant video frames (relevant video frames refer to video frames with corresponding identified targets) is obtained. and The heading angle calculation formula is: The preset number may be set according to actual needs, for example, to obtain the average speed of the first five frames.

[0101] It should be noted that if the number of the most recent relevant video frames is less than the preset number, the average speed of all existing relevant video frames may be obtained. For example, if there are only three relevant video frames, the average speed of the three relevant video frames may be obtained.

[0102] Step S130, based on the similarity of the motion states between the targets, the original perception results corresponding to the cameras are subjected to fusion tracking of the identified targets to obtain the current fusion perception result.

[0103] The fusion tracking process of identifying targets between multiple cameras may include three parts: preprocessing, fusion tracking and target management. Based on this, in some possible embodiments, the above step S130 may include the following sub-steps S131 to S133:

[0104] Sub-step S131, pre-processing the original perception results corresponding to each camera to obtain multiple pre-processed perception results.

[0105] In a possible implementation, the candidate perception results corresponding to the synchronous video frames can be obtained according to the timestamps in the original perception results corresponding to each camera; the candidate perception results are normalized in the coordinate system and the recognition targets in the preset region of interest are screened to obtain the pre-processed perception results. It should be noted that the normalization of the coordinate system is not a necessary step, as long as the coordinate system corresponding to the preset region of interest is consistent with the coordinate system corresponding to the original perception results, that is, in some embodiments, the normalization process of the coordinate system may not be performed.

[0106] When preprocessing the raw perception results, video frames with similar time (i.e., video frames with a time difference less than the frame interval) can be considered as synchronized frames based on the timestamp of each raw perception result. The raw perception results corresponding to each camera are unified into the UTM coordinate system, and the identified targets outside the ROI range are filtered out; where ROI (Region of Interest) refers to the area outlined from the processed image that requires special attention and processing. These areas can be marked by boxes, circles, or polygons for further image analysis or processing. The ROI range is pre-set and can be the area within the intersection road, that is, it is necessary to filter out targets outside the road and at a distance.

[0107] In specific implementation, the video streams output by each camera can be cached. When the time difference corresponding to the timestamps of the current video frames output by each camera is less than the frame interval (such as 30ms), these video frames are considered to be synchronized frames. If the video frame output by a camera is not a synchronized frame with the current video frame to be processed, you can wait for a preset time to obtain the synchronized frame corresponding to the camera, and then perform subsequent processing.

[0108] Sub-step S132, based on the motion state similarity between the targets and the stored previous fusion perception result, target matching and target fusion are performed on each pre-processed perception result to obtain a current fusion result.

[0109] The above-mentioned previous fusion perception result is the fusion perception result corresponding to the previous video frame. In a possible implementation method, the current fusion perception result can be predicted according to the previous fusion perception result to obtain the current predicted perception result; the recognition targets in each preprocessed perception result are associated with the fusion targets in the current predicted perception result by target identification to obtain a preliminary matching result; it is determined whether there are unassociated recognition targets in the preliminary matching result; if there are no unassociated recognition targets, the preliminary matching result is directly determined as the final matching result; if there are unassociated recognition targets, the unassociated recognition targets in the preliminary matching result are associated with the fusion targets in the current predicted perception result by target association of Mahalanobis distance based on motion state information to obtain the final matching result; the recognition targets that are successfully associated in the final matching result are fused through the pre-established uniform acceleration motion model and Kalman filter to obtain the current fusion result.

[0110] Target association is used to match the identified targets in the current video frame of multiple cameras with the current predicted perception results predicted based on the previous fusion perception results. In this embodiment, two methods are used for target association. One is to associate based on the target ID of the original camera. The camera output result includes the target ID of the identified target. The target ID output by the original camera in the previous frame is retained in each fusion target. When associating with the identified target of a new frame, the associated target can be quickly matched according to the target ID of the original camera in the previous frame. After ID association, calculate whether the distance between the position of the identified target in the perception result after preprocessing and the corresponding position in the current predicted perception result is less than the set threshold. If it is less than, the association is considered correct, otherwise the next matching method is used to associate the target. Among them, the threshold can be set according to actual needs and is not limited here.

[0111] Another target association method is to use Mahalanobis distance to characterize target similarity and solve the maximum match of the bipartite graph. Assuming that a camera outputs M identified targets and there are N fused targets in the current predicted perception result, the similarity between each identified target and each fused target is calculated to obtain an M×N similarity matrix; here, information such as target position, target speed, and target acceleration are used as features to calculate the Mahalanobis distance; finally, the Kuhn-Munkres algorithm is used to solve the similarity matrix to obtain the association result. Among them, the Mahalanobis distance (i.e., Mahalanobis distance) represents the distance between a point and a distribution. It is a method for calculating the similarity of two unknown sample sets. It takes into account the connection between various characteristics and is scale-independent; the Kuhn-Munkres algorithm is an algorithm for solving the optimal allocation problem, also known as the Hungarian algorithm, which is mainly used to find the maximum weighted complete match in a weighted bipartite graph.

[0112] A possible application scenario is as follows: after a target passes through two cameras, only the identified targets in these two cameras are fused. When the target enters the third camera, the identified target in the third camera is associated with the fused target based on the Mahalanobis distance. This two-level association method reduces the amount of computation required to solve the Mahalanobis distance and Kuhn-Munkres algorithm.

[0113] After the target is associated, the successfully associated targets are fused to establish a uniform acceleration motion model and Kalman filter algorithm. The Kalman filter state vector is defined as ,in , is the target location, θ is the heading angle, v x , v y For speed, v θ is the angular velocity of the heading angle, a x , a y is the acceleration; the measurement vector is defined as First, use the Kalman filter prediction step to get the predicted state vector corresponding to the current video frame , and then use the predicted state vector and the measurement vector corresponding to the current video frame to associate the target. Each fusion target may be associated with multiple recognition targets, and the measurement vector takes the average value of all successfully associated recognition targets.

[0114] Sub-step S133, performing target management on the current fusion result to obtain a fusion perception result; wherein the target management includes creating a new target and / or deleting a disappeared target.

[0115] Specifically, when there is a first target that meets a preset new target condition in the current fusion result, a fusion target corresponding to the first target is created; wherein the new target condition includes a successful association with a fusion target that has not corresponded to a previous fusion perception result.

[0116] When there is a second target that meets the preset target disappearance condition in the current fusion result, the second target is deleted; wherein the target disappearance condition includes that the number of consecutive tracking successes corresponding to the newly created fusion target does not reach the first preset number, or the number of consecutive tracking failures corresponding to the fusion target reaches the second preset number. The second preset number can be greater than the first preset number, and the values ​​of the first preset number and the second preset number can be set according to actual needs, which are not limited here.

[0117] The target management part is used to create new targets and delete lost targets. Taking the first preset number of times as 3 and the second preset number of times as 5 as an example, if the identified target is not successfully associated with the fused target, it can be considered that the target is a new target. If the new target can be correctly tracked for 3 consecutive frames, it is considered that the target really exists, otherwise it is deleted; if the fused target association fails, and fails to be successfully associated for 5 consecutive frames, it is determined that the fused target disappears and is deleted.

[0118] The structured information of the final output target includes: timestamp, global target ID (i.e. fused target ID), target type, position, velocity, acceleration and heading angle, etc.

[0119] The multi-camera fusion perception method provided by the embodiment of the present invention can obtain the current video frames output by multiple cameras deployed at the same intersection; perform 2D target recognition and tracking and target motion state estimation on the current video frames output by each camera to obtain the original perception results corresponding to each camera; wherein the original perception results include timestamp, target identification and motion state information; based on the motion state similarity between the targets, the original perception results corresponding to each camera are fused and tracked to identify the target to obtain the current fused perception results. In this way, target fusion between multiple cameras at the intersection is realized, a unified target perception result can be obtained, and the target motion state estimation accuracy is improved.

[0120] For ease of understanding, refer to Figures 2 to 4 The above-mentioned multi-camera fusion perception method is introduced.

[0121] The embodiment of the present invention proposes a multi-camera fusion perception method for an intersection. First, for each camera, an image recognition model and a tracking algorithm are used to implement 2D detection and tracking of the target in the video frame image, and the position of the 2D detected target in the 3D space is calculated. When estimating the position, a pixel point is selected as the 2D reference point of the target and the 3D position of the target is calculated according to the calibration parameters; however, due to the different positions and viewing angles of the target, the 2D reference point deviates from the actual position of the target; in addition, the 2D reference points of the same target in different cameras may correspond to different positions of the target, which has a great impact on the subsequent multi-camera fusion. The embodiment of the present invention proposes an adaptive target position estimation method, which can automatically adjust the 2D reference point according to the target position and the camera viewing angle to make it closer to the actual position of the target, reduce the error between different cameras, and combine the Kalman filter to reduce the 3D position drift caused by the jitter of the target's 2D detection frame.

[0122] On the basis of each camera identifying the target, the target information is synchronized in time and converted to a unified coordinate system. The recognition results of the same target in different cameras are found and fused according to the similarity of the motion states between all targets, so as to achieve the unification and continuous tracking of the target between different cameras. Since the recognition result of each camera includes the target ID, the embodiment of the present invention proposes a two-level target association method based on the target ID and Mahalanobis distance of the original camera, which can reduce the calculation amount of target association and improve the fusion speed.

[0123] The embodiment of the present invention can realize target fusion between multiple cameras at an intersection, thereby achieving continuous target tracking and improving the estimation accuracy of the target motion state.

[0124] See also Figure 2 The schematic diagram of the implementation principle of a multi-camera fusion perception method shown in FIG. 1 includes three main parts: 1. 2D target recognition and tracking module, 2. target motion state estimation module and 3. multi-camera fusion module. The following is a detailed introduction to these three parts.

[0125] 1. 2D target recognition and tracking module

[0126] The 2D target recognition and tracking module first pulls the RTSP video stream of each camera (such as camera 1, camera 2, camera 3 and camera 4) to obtain video frames, and records the current timestamp of each frame of video for time synchronization of multi-camera fusion.

[0127] In order to detect targets in multiple camera videos in real time, the Yolov5s model is used in the embodiment of the present invention to detect targets such as pedestrians, motor vehicles and non-motor vehicles. The target recognition information output by the 2D detection algorithm may include the target 2D frame, target type and confidence score, etc.

[0128] After the Yolov5s model obtains the target 2D frame, it uses the image multi-target tracking algorithm to track the target in the image. In the embodiment of the present invention, the ByteTrack algorithm is used as the 2D multi-target tracking algorithm, and the ByteTrack algorithm can achieve real-time operation. The structured information of the identified target finally output by the 2D tracking algorithm may include: timestamp, target ID, target type, and 2D detection frame, etc.

[0129] 2. Target motion state estimation module

[0130] This module calculates the target motion state based on the structured information of the target output by the 2D target recognition and tracking module. The module is divided into two steps: position estimation and Kalman filtering.

[0131] Position calculation first requires calibration of the camera parameters to obtain the conversion matrix from the image pixel coordinates to the actual space point coordinates H , which is a homography matrix. The spatial point coordinates can use the longitude and latitude obtained from GPS devices or high-precision maps. In order to facilitate subsequent calculations, the longitude and latitude are uniformly converted to the coordinates of the UTM coordinate system.

[0132] Target position calculation Select a 2D reference point from the 2D detection box ( u,v ), according to the formula The 3D position of the target (i.e. the original 3D position) can be calculated. Figure 3 As shown, the target can be regarded as a 3D bounding box (hereinafter referred to as 3D box), and the ideal reference point for calculating the 3D position is the center point of the bottom surface of the 3D box. However, this point is difficult to obtain directly based on the 2D detection frame. There is a certain error in selecting the center point of the bottom edge of the 2D detection frame or the center point of the 2D detection frame, so the reference point should be as close to the ideal reference point as possible.

[0133] According to the principles of projective geometry, we first obtain an intersection point based on the extension lines of parallel lines such as lane lines and curbs in the image. , the intersection is called the vanishing point; the target is located on the ground, the center of the bottom surface of the 3D box The point must be below the vanishing point, and the closer the distance is, the closer it is to the center of the 2D detection frame. , the farther away it is, the closer it is to the bottom center of the 2D detection frame , according to this relationship, the reference point is adaptively optimized. The image coordinate system is established with the upper left corner of the image as the origin of the coordinate system. Assume that the width and height of the image are W and H , the center point of the 2D detection box , the width and height of the 2D detection box are w and h , the 2D reference point is calculated as follows:

[0134] ;

[0135] in, δ ∈(0,1) is the scaling scale, which is set according to the camera’s installation viewing angle. The resulting 2D reference point is closer to the center of the 2D detection frame when the target is close to the camera, and closer to the center of the bottom edge of the 2D detection frame when the distance is far.

[0136] After obtaining the 2D reference point, the 3D position of the target can be calculated. In order to reduce the 3D position drift caused by the jitter of the 2D detection frame, a uniform acceleration motion model and Kalman filter are established to filter the 3D position in 3D space and obtain the velocity, acceleration and heading angle information. The position, velocity and acceleration information after Kalman filtering are more consistent with the actual state of the target, reducing the problem of position jitter and jump. The target heading angle can be calculated using the average value of the velocity obtained by Kalman filtering, and the average velocity of the previous 5 frames is obtained. and The heading angle calculation formula is: .

[0137] The final target motion state estimation module outputs the structured information of the target including: timestamp, target ID, target type, position, velocity, acceleration and heading angle.

[0138] 3. Multi-camera fusion module

[0139] The multi-camera fusion module includes three parts: preprocessing, fusion tracking and target management.

[0140] First, the original perception results are preprocessed, such as Figure 4 As shown in the figure, the preprocessing part includes three steps: time synchronization, coordinate conversion, and ROI filtering. Specifically, according to the timestamp of each original perception result, video frames with similar time are considered to be synchronized frames; the original perception results corresponding to each camera are unified into the UTM coordinate system, and the identified targets outside the ROI range are filtered out. The ROI is set to the area within the intersection road, and the targets outside the road and far away are filtered out.

[0141] Fusion tracking includes target association and filtering. Target association is used to match the identified targets in the current video frame of multiple cameras with the current predicted perception results predicted based on the previous fusion perception results. In this embodiment, two methods are used for target association. One is to associate based on the target ID of the original camera. The camera output result includes the target ID of the identified target. The target ID output by the original camera in the previous frame is retained in each fusion target. When associating with the identified target of a new frame, the associated target can be quickly matched according to the target ID of the original camera in the previous frame. After ID association, calculate whether the distance between the position of the identified target in the perception result after preprocessing and the corresponding position in the current predicted perception result is less than the set threshold. If it is less than, the association is considered correct, otherwise the next matching method is used to associate the target. Among them, the threshold can be set according to actual needs and is not limited here.

[0142] Another target association method is to use Mahalanobis distance to characterize target similarity and solve the maximum matching of the bipartite graph. Assuming that a camera outputs M identified targets and there are N fused targets in the current predicted perception result, the similarity between each identified target and each fused target is calculated to obtain an M×N similarity matrix; here, information such as target position, target speed, and target acceleration are used as features to calculate the Mahalanobis distance; finally, the Kuhn-Munkres algorithm is used to solve the similarity matrix to obtain the association result.

[0143] After the target is associated, the successfully associated targets are fused to establish a uniform acceleration motion model and Kalman filter algorithm. The Kalman filter state vector is defined as ,in , is the target location, θ is the heading angle, v x , v y For speed, v θ is the angular velocity of the heading angle, a x , a y is the acceleration; the measurement vector is defined as First, the Kalman filter prediction step is used to obtain the predicted state vector corresponding to the current video frame, and then the predicted state vector and the measurement vector corresponding to the current video frame are used to associate the target. Each fusion target may be associated with multiple recognition targets, and the measurement vector takes the average value of all successfully associated recognition targets.

[0144] The target management part is used to create new targets and delete lost targets. If the identified target is not successfully associated with the fused target, it can be considered as a new target. If the new target can be correctly tracked for 3 consecutive frames, it is considered that the target really exists, otherwise it is deleted; if the fused target association fails, and fails to be successfully associated for 5 consecutive frames, it is determined that the fused target disappears and is deleted.

[0145] like Figure 4As shown, for the fusion tracking and target management part: after preprocessing, first perform ID association (i.e. target association based on target identification) to determine whether the ID association is successful. If yes (i.e. ID association is successful), then perform Kalman filter update (need to combine the predicted state vector corresponding to the current video frame, which is obtained by performing Kalman filter prediction on the multi-camera fusion result corresponding to the previous video frame), so as to obtain the multi-camera fusion result. If no (i.e. ID association fails), then perform distance association (i.e. target association based on Mahalanobis distance) to determine whether the distance association is successful. If yes (i.e. distance association is successful), then perform Kalman filter update to obtain the multi-camera fusion result; if no (i.e. distance association fails), when the detection does not match the tracking (i.e. the identification target is not associated with the fusion target), perform target management of the new target; when the tracking does not match the detection (i.e. the fusion target is not associated with the identification target, and the fusion target tracking fails), perform target management of the message target, so as to obtain the multi-camera fusion result (i.e. the current fusion perception result). After obtaining the multi-camera fusion result, the fusion output can be performed (i.e. the output of the current fusion perception result), and the multi-camera fusion result will also be used for the Kalman filter prediction of the next video frame.

[0146] The final output target structured information includes: timestamp, global target ID (i.e. fused target ID), target type, position, velocity, acceleration and heading angle, etc. The proposed two-level association method reduces the computational complexity of the Mahalanobis distance and Kuhn-Munkres algorithm.

[0147] The embodiments of the present invention have at least the following beneficial effects:

[0148] (1) An adaptive target position estimation method is proposed to reduce the position estimation error caused by the difference between the target position and the camera perspective, reduce the jitter of the position and speed estimation, and better conform to the actual motion state of the target.

[0149] (2) The target fusion between multiple cameras at the intersection is realized, which can obtain a unified target perception result and improve the accuracy of target motion state estimation. The two-level target association method based on the original camera target ID and Mahalanobis distance is used to reduce the association calculation amount and can run in real time.

[0150] Corresponding to the above-mentioned multi-camera fusion perception method, the embodiment of the present invention also provides a multi-camera fusion perception system. Figure 5 A structural schematic diagram of a multi-camera fusion perception system is shown, the system comprising:

[0151] An acquisition module 501 is used to acquire current video frames output by multiple cameras deployed at the same intersection;

[0152] The perception module 502 is used to perform 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera, and obtain the original perception result corresponding to each camera; wherein the original perception result includes a timestamp, a target identifier and motion state information;

[0153] The fusion module 503 is used to perform fusion tracking of the identified target on the original perception results corresponding to each camera based on the similarity of the motion states between the targets, so as to obtain the current fusion perception result.

[0154] The multi-camera fusion perception system provided by the embodiment of the present invention can obtain the current video frames output by multiple cameras deployed at the same intersection; perform 2D target recognition and tracking and target motion state estimation on the current video frames output by each camera to obtain the original perception results corresponding to each camera; wherein the original perception results include timestamp, target identification and motion state information; based on the motion state similarity between the targets, the original perception results corresponding to each camera are fused and tracked to identify the target to obtain the current fused perception results. In this way, target fusion between multiple cameras at the intersection is achieved, a unified target perception result can be obtained, and the target motion state estimation accuracy is improved.

[0155] Furthermore, the perception module 502 is specifically used to: perform 2D target recognition and tracking on the current video frame output by each camera through a preset 2D detection algorithm and a 2D tracking algorithm, and obtain the 2D structured information of the recognized target corresponding to each camera; wherein the 2D structured information of the recognized target includes a timestamp, a target identifier, and a 2D detection frame; according to the 2D structured information of the recognized target corresponding to each camera and the preset installation viewing angle information, determine the 2D reference point of the recognized target in the corresponding current video frame; wherein the 2D reference point of the recognized target decreases as the distance between the recognized target and the camera decreases. The center point of the bottom edge of the 2D detection frame of the identified target gradually becomes the center point of the 2D detection frame close to the identified target; according to the preset transformation matrix corresponding to each camera and the 2D reference point of the identified target in the corresponding current video frame, the original 3D position information of the identified target corresponding to each camera is determined; wherein the original 3D position information of the identified target is used to indicate the actual position of the identified target; through the pre-established uniform acceleration motion model and Kalman filter, the original 3D position information of the identified target corresponding to each camera is filtered, and the speed, acceleration and heading angle are obtained to obtain the original perception result corresponding to each camera.

[0156] Furthermore, the perception module 502 is also used to: determine the center point coordinates and height of the 2D detection frame of the identified target according to the 2D structured information of the identified target corresponding to each camera; and calculate the 2D reference point of the identified target by the following formula:

[0157] ;

[0158] in, u ref_2D Indicates the horizontal coordinate of the 2D reference point, v ref_2D Indicates the ordinate of the 2D reference point, u c Represents the horizontal coordinate of the center point of the 2D detection box, v c Indicates the ordinate of the center point of the 2D detection box, v v Indicates the vertical coordinate of the preset vanishing point corresponding to the camera, h Indicates the height of the 2D detection box, H Indicates the image height of the current video frame. δ Indicates the zoom scale corresponding to the preset installation viewing angle information of the camera.

[0159] Furthermore, the above-mentioned fusion module 503 is specifically used for: preprocessing the original perception results corresponding to each camera to obtain multiple preprocessed perception results; performing target matching and target fusion on each preprocessed perception result based on the motion state similarity between the targets and the stored previous fusion perception result to obtain the current fusion result; performing target management on the current fusion result to obtain the fused perception result; wherein the target management includes creating new targets and / or deleting disappeared targets.

[0160] Furthermore, the above-mentioned fusion module 503 is also used to: obtain candidate perception results corresponding to the synchronized video frame according to the timestamp in the original perception results corresponding to each camera; perform coordinate system normalization processing on the candidate perception results and screen the identification targets in the preset area of ​​interest to obtain the pre-processed perception results.

[0161] Furthermore, the above-mentioned fusion module 503 is also used to: predict the current fused perception result according to the previous fused perception result to obtain the current predicted perception result; perform target association of target identification on the recognized targets in each preprocessed perception result and the fused target in the current predicted perception result to obtain a preliminary matching result; determine whether there are unassociated recognized targets in the preliminary matching result; if there are unassociated recognized targets, perform target association of Mahalanobis distance on the unassociated recognized targets in the preliminary matching result and the fused target in the current predicted perception result based on the motion state information to obtain the final matching result; fuse the recognized targets that are successfully associated in the final matching result through the pre-established uniform acceleration motion model and Kalman filter to obtain the current fusion result.

[0162] Furthermore, the above-mentioned fusion module 503 is also used for: when there is a first target that meets the preset new target condition in the current fusion result, creating a fusion target corresponding to the first target; wherein the new target condition includes not being successfully associated with the fusion target corresponding to the previous fusion perception result; when there is a second target that meets the preset disappeared target condition in the current fusion result, deleting the second target; wherein the disappeared target condition includes that the number of continuous tracking successes corresponding to the newly created fusion target does not reach the first preset number, or the number of continuous tracking failures corresponding to the fusion target reaches the second preset number.

[0163] The multi-camera fusion perception system provided in this embodiment has the same implementation principle and technical effects as those of the aforementioned multi-camera fusion perception method embodiment. For the sake of brief description, for matters not mentioned in the multi-camera fusion perception system embodiment, reference may be made to the corresponding contents in the aforementioned multi-camera fusion perception method embodiment.

[0164] like Figure 6 As shown, an electronic device 600 provided by an embodiment of the present invention includes: a processor 601, a memory 602 and a bus, the memory 602 stores a computer program that can be run on the processor 601, when the electronic device 600 is running, the processor 601 and the memory 602 communicate through the bus, and the processor 601 executes the computer program to implement the above-mentioned multi-camera fusion perception method.

[0165] Specifically, the memory 602 and the processor 601 can be general-purpose memories and processors, which are not specifically limited here.

[0166] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the multi-camera fusion perception method in the above method embodiment is executed. The computer-readable storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disk, etc., various media that can store program codes.

[0167] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0168] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0169] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0170] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or modules, which can be electrical, mechanical or other forms.

[0171] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0174] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0175] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0176] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera fusion perception method, characterized in that: include: Get the current video frames output by multiple cameras deployed at the same intersection; Perform 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera to obtain the original perception result corresponding to each camera; wherein the original perception result includes a timestamp, a target identifier and motion state information; Based on the similarity of the motion states between the targets, the original perception results corresponding to the cameras are subjected to fusion tracking of the identified targets to obtain the current fusion perception results; The performing 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera to obtain the original perception result corresponding to each camera includes: By using a preset 2D detection algorithm and a 2D tracking algorithm, 2D target recognition and tracking are performed on the current video frame output by each camera to obtain 2D structured information of the recognized target corresponding to each camera; wherein the 2D structured information of the recognized target includes a timestamp, a target identifier and a 2D detection frame; Determine the 2D reference point of the identified target in the corresponding current video frame according to the 2D structural information and the preset installation viewing angle information of the identified target corresponding to each camera; wherein the 2D reference point of the identified target gradually changes from the center point of the bottom edge of the 2D detection frame close to the identified target to the center point of the 2D detection frame close to the identified target as the distance between the identified target and the camera decreases; Determine the original 3D position information of the identified target corresponding to each camera according to the preset transformation matrix corresponding to each camera and the 2D reference point of the identified target in the corresponding current video frame; wherein the original 3D position information of the identified target is used to indicate the actual position of the identified target; Through the pre-established uniform acceleration motion model and Kalman filter, the original 3D position information of the identified target corresponding to each camera is filtered, and the speed, acceleration and heading angle are obtained to obtain the original perception result corresponding to each camera.

2. The multi-camera fusion perception method according to claim 1, characterized in that: The determining of the 2D reference point of the identified target in the corresponding current video frame according to the 2D structured information of the identified target and the preset installation viewing angle information corresponding to each camera includes: Determine the center point coordinates and height of the 2D detection frame of the identified target according to the 2D structured information of the identified target corresponding to each camera; The 2D reference point of the identified target is calculated by the following formula: ； in, u ref_2D Indicates the horizontal coordinate of the 2D reference point, v ref_2D Indicates the ordinate of the 2D reference point, u c Represents the horizontal coordinate of the center point of the 2D detection box, v c Indicates the ordinate of the center point of the 2D detection box, v v Indicates the vertical coordinate of the preset vanishing point corresponding to the camera, h Indicates the height of the 2D detection box, H Indicates the image height of the current video frame. δ Indicates the zoom scale corresponding to the preset installation viewing angle information of the camera.

3. The multi-camera fusion perception method according to claim 1, characterized in that: Based on the similarity of the motion states between the targets, the original perception results corresponding to the cameras are subjected to fusion tracking of the identified targets to obtain the current fusion perception results, including: Preprocessing the original perception results corresponding to each of the cameras to obtain a plurality of preprocessed perception results; Based on the motion state similarity between the targets and the stored previous fusion perception result, target matching and target fusion are performed on each of the preprocessed perception results to obtain a current fusion result; wherein the previous fusion perception result is a fusion perception result corresponding to a previous video frame; Performing target management on the current fusion result to obtain the fusion perception result; wherein the target management includes creating a new target and / or deleting a disappeared target; The target matching and target fusion are performed on each of the pre-processed perception results based on the motion state similarity between the targets and the stored previous fusion perception result to obtain the current fusion result, including: Predicting the current fused perception result according to the previous fused perception result to obtain a current predicted perception result; Performing target association of target identification on the identified target in each of the preprocessed perception results and the fused target in the current predicted perception result to obtain a preliminary matching result; Determining whether there is an unassociated identification target in the preliminary matching result; If there is an unassociated recognition target, the unassociated recognition target in the preliminary matching result is associated with the fusion target in the current predicted perception result by using the motion state information as a feature, to obtain a final matching result; The successfully associated identified targets in the final matching result are fused through the pre-established uniform acceleration motion model and Kalman filter to obtain the current fusion result.

4. The multi-camera fusion perception method according to claim 3 is characterized in that: The preprocessing of the original perception results corresponding to each of the cameras to obtain a plurality of preprocessed perception results includes: Obtaining candidate perception results corresponding to the synchronized video frames according to the timestamps in the original perception results corresponding to the cameras; The candidate perception results are subjected to coordinate system normalization processing and identification target screening in a preset region of interest to obtain the preprocessed perception results.

5. The multi-camera fusion perception method according to claim 3 is characterized in that: The target management of the current fusion result to obtain the fusion perception result includes: When there is a first target that meets a preset new target condition in the current fusion result, creating a fusion target corresponding to the first target; wherein the new target condition includes not being successfully associated with the fusion target corresponding to the previous fusion perception result; When there is a second target that meets the preset target disappearance condition in the current fusion result, the second target is deleted; wherein the target disappearance condition includes that the number of continuous tracking successes corresponding to the newly created fusion target does not reach a first preset number, or the number of continuous tracking failures corresponding to the fusion target reaches a second preset number.

6. A multi-camera fusion perception system, characterized in that: include: An acquisition module is used to acquire current video frames output by multiple cameras deployed at the same intersection; A perception module, used to perform 2D target recognition and tracking and target motion state estimation on the current video frame output by each camera, and obtain the original perception result corresponding to each camera; wherein the original perception result includes a timestamp, a target identifier and motion state information; A fusion module is used to perform fusion tracking of the identified target on the original perception results corresponding to each camera based on the similarity of the motion states between the targets, so as to obtain the current fusion perception result; The perception module is specifically used to: perform 2D target recognition and tracking on the current video frame output by each camera through a preset 2D detection algorithm and a 2D tracking algorithm, and obtain 2D structured information of the recognized target corresponding to each camera; wherein the 2D structured information of the recognized target includes a timestamp, a target identifier, and a 2D detection frame; determine the 2D reference point of the recognized target in the corresponding current video frame according to the 2D structured information of the recognized target corresponding to each camera and the preset installation viewing angle information; wherein the 2D reference point of the recognized target decreases from closer to the recognized target as the distance between the recognized target and the camera decreases. The center point of the bottom edge of the 2D detection frame gradually becomes the center point of the 2D detection frame close to the identified target; according to the preset transformation matrix corresponding to each camera and the 2D reference point of the identified target in the corresponding current video frame, the original 3D position information of the identified target corresponding to each camera is determined; wherein the original 3D position information of the identified target is used to indicate the actual position of the identified target; through the pre-established uniform acceleration motion model and Kalman filter, the original 3D position information of the identified target corresponding to each camera is filtered, and the speed, acceleration and heading angle are obtained to obtain the original perception result corresponding to each camera.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the multi-camera fusion perception method described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-camera fusion perception method described in any one of claims 1 to 5 is executed.

Images