A method for target fusion and blind spot filling in multi-camera detection

Abstract

The present invention discloses a method for multi-camera target fusion and blind spot filling, comprising installing multiple cameras at an intersection, acquiring multiple video source images, and establishing a pixel coordinate system; performing target detection on each of the multiple video source images to obtain the target detection type; obtaining the pixel coordinates of the target pixel point in the pixel coordinate system for each detected target based on the target detection type and the characteristics of the box, and converting them to obtain the target coordinates; defining a specific area and a region of interest for each camera, and directly outputting all targets in the specific area; establishing a custom coordinate system, and performing a bird's-eye perspective transformation on the target coordinates in the region of interest to obtain bird's-eye coordinates; performing target fusion based on the bird's-eye coordinates, and outputting all targets in the region of interest. The present invention avoids the errors caused by fusion based solely on the surface features of the target box image, and does not require joint calibration of the video source images, thereby reducing a significant amount of time and labor costs.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic detection, and in particular to a method for target fusion and blind spot filling in multi-camera detection. Background Art

[0002] Smart transportation plays a vital role in achieving sustainable smart city development and economic growth. With rising per capita economic levels, the number of motor vehicles, non-motor vehicles, and pedestrians in traffic scenarios is increasing, making manual traffic flow and traffic violation statistics a thing of the past. Currently, there are high demands for real-time roadside perception, violation alarms, and evidence preservation. Solutions using multiple roadside cameras to collaboratively detect targets are often used to compensate for blind spots on the road, improving target detection recall and reducing missed detection rates.

[0003] Existing methods for fusing multi-camera detection results include:

[0004] 1. Based on extracting target features such as color, texture, shape, etc., and using these features for fusion.

[0005] 2. Target fusion based on deep learning: By jointly training multiple camera images to obtain a deep learning model, high-level semantic features in the image are extracted and these features are used to achieve target fusion.

[0006] The above method has at least the following disadvantages:

[0007] 1. Target features are extracted using color, texture, and shape. Since multiple cameras in roadside projects are usually installed at the intersection, shooting from four directions toward the center of the intersection, there are slight differences in the texture and shape of the target due to different shooting angles. Furthermore, the cameras are affected by the direction of light, so the target colors captured by cameras installed in different directions may differ slightly, reducing the accuracy of target fusion.

[0008] 2. Before jointly training images from multiple cameras, the image data from multiple cameras needs to be jointly calibrated, which brings about a large amount of time and manpower costs. Summary of the Invention

[0009] The purpose of the present invention is to provide a method for multi-camera target fusion and blind spot filling, which avoids the error caused by fusion only through the surface features of the target frame image, and does not require joint calibration of the video source images, thereby reducing a lot of time and labor costs.

[0010] The embodiments of the present invention are achieved through the following technical solutions:

[0011] A method for multi-camera target fusion and blind spot filling, characterized by comprising the following steps:

[0012] Install multiple cameras at the intersection to obtain multiple video source images and establish a pixel coordinate system;

[0013] Perform target detection on multiple video source images respectively to obtain target detection types;

[0014] According to the target detection type and box characteristics, the pixel coordinates of each detected target pixel are obtained in the pixel coordinate system and converted to the target coordinates;

[0015] Define the specific area and area of ​​interest for each camera, and directly output all targets in the specific area; establish a custom coordinate system, and perform bird's-eye perspective transformation on the target coordinates in the area of ​​interest to obtain bird's-eye coordinates;

[0016] Target fusion is performed based on bird's-eye view coordinates, and all targets in the area of ​​interest are output.

[0017] In one embodiment of the present invention, the target detection types include motor vehicle targets and non-motor vehicle targets.

[0018] In one embodiment of the present invention, the specific method of obtaining the pixel coordinates of each detected target pixel in the pixel coordinate system and converting them to obtain the target coordinates according to the target detection type and box characteristics is as follows:

[0019] When the target is a non-motor vehicle, there is a conversion formula:

[0020]

[0021] Where x and y are the target coordinates, x0, is the pixel coordinate, and h is the height of the box.

[0022] In one embodiment of the present invention, the specific method of obtaining the pixel coordinates of each detected target pixel in the pixel coordinate system and converting them to obtain the target coordinates according to the target detection type and box characteristics is as follows:

[0023] When the target belongs to a motor vehicle target and satisfies the relation When , there is a conversion formula:

[0024]

[0025] When the target belongs to a motor vehicle target and satisfies the relation When , a threshold is set for the aspect ratio of the box;

[0026] When the aspect ratio of the target box is greater than the threshold, the conversion formula is:

[0027]

[0028] When the aspect ratio of the target box is less than the threshold, there is a conversion formula:

[0029]

[0030] In the above formula, x and y are the target coordinates, x0, is the pixel coordinate, h is the height of the box, w is the width of the box, and H is the height of a single frame image.

[0031] In one embodiment of the present invention, the specific formula for performing bird's-eye perspective transformation on the target coordinates in the region of interest to obtain the bird's-eye view coordinates is:

[0032]

[0033] Where u' and v' are bird's-eye view coordinates, k 11 、k 12 、k 13 、k 21 、k 22 、k 23 、k 31 、k 32 is the coefficient, and x and y are the target coordinates.

[0034] In one embodiment of the present invention, the specific method of performing target fusion based on bird's-eye view coordinates and outputting all targets in the region of interest is as follows:

[0035] Set the distance threshold D min and D max , traverse the camera S with the largest number of targets in the area of ​​interest kmax and other cameras S i All goals;

[0036] Find camera S kmax Each target in the region of interest is i The target d with the closest Euclidean distance in the region of interest min ;

[0037] Compare d min With D min and D max The size of the value, if it satisfies the relationship d min >D max , it means that the two are not the same target; if the relationship d min <D min, it means that the two are the same goal and are integrated; if the relationship D is satisfied min <d min <D max , then determine whether the target types of the two are consistent. If so, they are the same target and are fused. If not, they are fused again in the next frame of video source image.

[0038] In one embodiment of the present invention, the specific method of performing target fusion based on bird's-eye view coordinates and outputting all targets in the region of interest further includes:

[0039] Set the fusion threshold. If the fusion times exceed the fusion threshold and the fusion fails, the fusion will be stopped.

[0040] The technical solutions of the embodiments of the present invention have at least the following advantages and beneficial effects:

[0041] The present invention uses a joint deployment scheme of multiple cameras to compensate for blind spots caused by mutual occlusion between targets in road target detection scenarios, reduce the missed detection rate of target detection, avoid the decline in target detection accuracy due to factors such as lighting and weather, and improve the robustness of target detection; at the same time, by fusing image information from multiple cameras, the position and motion trajectory of the target can be tracked more accurately, the continuity of target tracking is enhanced, and the accuracy of target recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the steps of the present invention;

[0043] Figure 2 This is the installation plan diagram of the intersection camera;

[0044] Figure 3 A schematic diagram of the intersection is captured for the camera;

[0045] Figure 4 Schematic diagram of target detection type;

[0046] Figure 5 Schematic diagram of bird's-eye view perspective transformation;

[0047] Figure 6 Schematic diagram of target fusion;

[0048] Figure 7 This is a schematic diagram of a custom coordinate system;

[0049] Figure 8 A schematic diagram of the pixel coordinate system. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0051] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0052] Example 1:

[0053] Please refer to Figures 1-8 ,A method for multi-camera detection target fusion and blind spot filling, comprising the following steps:

[0054] 100. Install multiple cameras at the intersection, acquire multiple video source images, and establish a pixel coordinate system;

[0055] 200. Perform target detection on multiple video source images respectively to obtain target detection types;

[0056] 300. According to the target detection type and the characteristics of the box, the pixel coordinates of each detected target pixel are obtained in the pixel coordinate system and converted to obtain the target coordinates;

[0057] 400. Define the specific area and the area of ​​interest of each camera, and directly output all targets in the specific area; establish a custom coordinate system, and perform a bird's-eye perspective transformation on the target coordinates in the area of ​​interest to obtain the bird's-eye view coordinates;

[0058] 500. Perform target fusion based on the bird's-eye view coordinates and output all targets in the area of ​​interest.

[0059] In step 100, the pixel coordinate system is established as follows Figure 8 As shown: A coordinate system is established with the upper left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis.

[0060] The specific method of step 200 is:

[0061] The target detection algorithm yolov8 is used to detect the target of the video source image, obtain the target detection type, and determine whether the target is a motor vehicle target or a non-motor vehicle target.

[0062] The specific method of step 300 is:

[0063] When the target is a non-motor vehicle, there is a conversion formula:

[0064]

[0065] Where x and y are the target coordinates, x0, is the pixel coordinate, and h is the height of the box.

[0066] When the target belongs to a motor vehicle target and satisfies the relation When , there is a conversion formula:

[0067]

[0068] When the target belongs to a motor vehicle target and satisfies the relation When , a threshold is set for the aspect ratio of the box;

[0069] When the aspect ratio of the target box is greater than the threshold, the conversion formula is:

[0070]

[0071] When the aspect ratio of the target box is less than the threshold, there is a conversion formula:

[0072]

[0073] In the above formula, x and y are the coordinates of the custom coordinate system, x0, is the coordinate of the pixel coordinate system, h is the height of the box, w is the width of the box, and H is the height of a single frame image.

[0074] The specific formula of step 400 is:

[0075]

[0076] Where u' and v' are bird's-eye view coordinates, k 11 、k 12 、k 13 、k 21 、k 22 、k 23 、k 31 、k 32 is the coefficient, and x and y are the target coordinates.

[0077] In step 400, the method for establishing the custom coordinate system is as follows: Figure 7As shown: Take the center point of the target detection area at the intersection, such as the center point of the middle area of ​​the intersection as the reference point to set a suitable coordinate system; for example, according to the size of the intersection and considering the width of the road surface on both sides of the intersection, select a suitable position as the coordinate origin, such as the intersection of the extension lines of the vertical lines of the two adjacent roads at the intersection as the coordinate origin, and set the directions of the coordinate x-axis and y-axis, as shown in the schematic diagram. Figure 7 The coordinate system Oxy shown;

[0078] It should be noted that in real scenes, the origin of the coordinate system Oxy often contains buildings, which makes it inconvenient to obtain the position of the calibration point selected on the image in the custom coordinate system. Therefore, when obtaining the corresponding point in the custom coordinate system, you can first obtain the coordinate origin of the point in the coordinate system. The coordinates below are as shown in the schematic diagram Figure 7 The coordinate system shown Then the coordinate value of the coordinate point in the coordinate system Oxy is calculated through translation.

[0079] The specific method of step 500 is:

[0080] Set the distance threshold D min and D max , traverse the camera S with the largest number of targets in the area of ​​interest kmax and other cameras S i All goals;

[0081] Find camera S kmax Each target in the region of interest is i The target d with the closest Euclidean distance in the region of interest min ;

[0082] Compare d min With D min and D max The size of the value, if it satisfies the relationship d min >D max , it means that the two are not the same target; if the relationship d min <D min , it means that the two are the same goal and are integrated; if the relationship D is satisfied min <d min <D max , then determine whether the target types of the two are consistent. If so, they are the same target and are fused. If not, they are fused again in the next frame of video source image.

[0083] In one embodiment of the present invention, the step of performing target fusion based on the target position after bird's-eye perspective transformation and outputting all targets and their types at the intersection further includes:

[0084] Set the fusion threshold. If the fusion times exceed the fusion threshold and the fusion fails, the fusion will be stopped.

[0085] Example 2:

[0086] This example is a detailed analysis of Example 1.

[0087] In step 100, the number of cameras is preferably four, and the installation method is as follows: Figure 2 As shown, the video source image is acquired in real time through the camera.

[0088] In step 200, a specific method of performing target detection on multiple video source images is to use the target detection algorithm yolov8 to obtain target detection types, wherein the obtained target detection types include motor vehicle targets and non-motor vehicle targets.

[0089] In step 300, if Figure 3 As shown in , for the target in the middle of the intersection, since it is perceived by multiple video source images, the perception result is the center point coordinates (x0, y0) of a target on the image corresponding to the video source image and the width (w) and height (h) of the box, as shown in Figure 4 Example of target detection results.

[0090] In order to obtain the pixel position of the target more accurately and make it more accurate when mapped to the custom coordinate system, a method based on the target type and the aspect ratio of the box is adopted to set the threshold. The target pixel position and box height are combined, and the information of the entire box is considered to select a pixel point as the pixel position of the target.

[0091] 1. For non-motorized vehicle targets, such as pedestrians, bicycles, and (electric) motorcycles, the ground occupancy and spatial occupancy of a single target are relatively small. The target pixel position is obtained using Formula 1, which takes the coordinates of the target box two pixels above the center point of the lower edge.

[0092]

[0093] -Formula 1

[0094] The pixel position at this time is not the coordinate of the center point of the lower edge of the box because when the data is labeled, the boundary of the box is composed of the minimum circumscribed rectangle of the detected target. Directly taking a pixel on the boundary as the target pixel position is not accurate enough.

[0095] 2. For motor vehicle targets, such as cars, buses, and trucks, individual targets have high ground and space occupancy rates. When selecting a pixel on an image as the target location, a threshold is set based on the target box's aspect ratio to provide calculation methods for the target pixel location (x, y) in different situations. In this embodiment, the box's aspect ratio threshold is set to 1.

[0096] (a) When the target satisfies the relation When , the target is considered to be at a farther position in the image field of view, and the target pixel position is obtained by the above formula 1. Indicates the y-coordinate value of the bottom edge of the box, and H represents the height of a single frame image.

[0097] (b) When the target satisfies the relation When, according to the target box frame aspect ratio ( ) value, divided into two cases:

[0098] I. When the target box meets the , it is assumed that the target is in a longitudinal moving state relative to the field of view, and is moving forward or backward (facing the front or rear of the motor vehicle target in the image). Considering the characteristics of the image, the target pixel position is obtained by formula 2.

[0099]

[0100] -Formula 2

[0101] At this time, the pixel position does not take the coordinates of the center point of the box, but adds the height parameter of the box. The pixel of the center point of the target box is translated downward by 1 / 4 of the height of the box, which effectively avoids the characteristic that the ground corresponding to the pixel position of the center point of the target box does not match the position of the target.

[0102] Ⅱ. When the target Box meets , the target is considered to be turning or moving sideways relative to the field of view (either side of the motor vehicle target can be seen in the image). In this case, the pixel position of the target is obtained using Formula 3. Formula 3 calculates the pixel position of a vehicle moving sideways relative to the field of view closer to the center point of the bottom edge of the box.

[0103]

[0104] -Formula 3

[0105] In this embodiment, in Formula 1, Formula 2 and Formula 3, x and y are target coordinates, x0, is the pixel coordinate, h is the height of the box, and w is the width of the box.

[0106] In step 400, the formula is derived as follows:

[0107] 1. Calibrate four sets of pixel coordinates and custom coordinates respectively, and select four appropriate pixel points as calibration points before perspective transformation.

[0108] 2. Then, at the intersection, based on the custom coordinate origin, coordinate axis direction, and reference point, determine the position of the calibration point selected on the image in the custom coordinate system as the calibration point after perspective transformation; specifically, the following steps are included:

[0109] (a) Customize the coordinate system at the intersection and take the center point of the target detection area, such as the center point of the middle area of ​​the intersection as the reference point to set the appropriate coordinate system. For example, based on the size of the intersection and considering the width of the road surface on both sides of the intersection, select a suitable location as the coordinate origin, such as the intersection of the extended lines of the vertical lines of the two adjacent roads at the intersection as the coordinate origin, and set the directions of the coordinate x-axis and y-axis. The schematic diagram is shown as follows: Figure 7 The coordinate system Oxy is shown.

[0110] (b) Define a suitable coordinate value for the reference point in the Oxy coordinate system. In real scenes, the origin of the coordinate system Oxy often contains buildings, which makes it inconvenient to obtain the position of the calibration point selected on the image in the custom coordinate system. Therefore, when obtaining the corresponding point in the custom coordinate system, you can first obtain the coordinate value of the point in the coordinate system. The coordinates of the coordinate point in the coordinate system Oxy are then calculated through translation.

[0111] 3. Calculate the perspective transformation matrix of multiple video sources for the four groups of one-to-one correspondence points calibrated for each video source image. Specifically, the following steps are included:

[0112] (a) Let A, B, C, and D be the four pixel calibration points on a video source image before perspective transformation.

[0113] (b) Let A', B', C', D' be the calibration points of A, B, C, D in the custom coordinate system; consider A', B', C', D' to be the points in the custom coordinate system after perspective transformation of A, B, C, D in the pixel coordinate system.

[0114] (c) Let M be the perspective transformation matrix, is the homogeneous coordinate of the original image pixel, is the transformed homogeneous coordinate, we have:

[0115]

[0116] We can get:

[0117]

[0118] Then we have:

[0119]

[0120] make , then:

[0121]

[0122] -Formula 4

[0123] Substituting the four groups of mapping points A, B, C, D and A', B', C', D' into formula 4, the perspective transformation matrix can be obtained by joint calculation.

[0124] In step 400, the ultimate goal of using multiple cameras to perform target detection at an intersection is to accurately output all targets and types within the effective detection range of the intersection. When the detection results are finally output, all targets are divided into two categories.

[0125] The first category involves targets that do not require fusion. Each camera defines a region of special (ROS) that serves as the effective detection range for that camera. Other cameras may not have this region of special coverage or have poor field of view, such as the lane outside a sidewalk. For ROS targets, detection is performed only by a single camera, so Hungarian optimization matching and fusion are not necessary.

[0126] The other type is targets that need to be fused after Hungarian matching before output; a region of interest (ROI) is defined for each camera, and the ROI serves as the effective detection range for multiple video sources, such as the area in the middle of an intersection.

[0127] For each camera ROI area target:

[0128] 1. First, count the number of targets k in each camera ROI area and record the maximum value k max Corresponding camera;

[0129] 2. Perform bird's-eye perspective transformation on the ROI area of ​​each camera, and transform all targets in the ROI area of ​​each camera into a custom coordinate system through bird's-eye perspective transformation.

[0130] In step 500, in the custom coordinate system, k max The corresponding camera (set to ) of the ROI are detected separately with other sources (set as , ) ROI detection targets are fused using the Hungarian optimization matching algorithm. The cost matrix is ​​calculated by the Euclidean distance between the targets in the ROI of different cameras in the custom coordinate system. At the same time, the distance threshold D needs to be set. min and D max , traverse the camera and Find the camera for all objects in the ROI Each target in the ROI is The target with the closest Euclidean distance in the upper ROI, and record the closest distance value d min :like , then it is believed that the two are not the same target; if , then it is believed that the two must be the same target, and the type of the corresponding target is determined by the side with the larger box area. We believe that the side with the larger box area is less likely to be blocked, and the accuracy of target recognition is higher; if , then consider whether the types corresponding to the two targets are consistent. If the target types are consistent, they are considered to be the same target. If the target types are inconsistent, they are fused again in the next frame of target detection data as the target to be fused. If the fusion fails for 10 consecutive frames, they will no longer be fused.

[0131] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for multi-camera target fusion and blind spot filling, characterized in that: The following steps are involved: Install multiple cameras at the intersection to obtain multiple video source images and establish a pixel coordinate system; Perform target detection on multiple video source images respectively to obtain target detection types; According to the target detection type and box characteristics, the pixel coordinates of each detected target pixel are obtained in the pixel coordinate system and converted to the target coordinates; Define the specific area and area of ​​interest for each camera, and directly output all targets in the specific area; establish a custom coordinate system, and perform bird's-eye perspective transformation on the target coordinates in the area of ​​interest to obtain bird's-eye coordinates; Perform target fusion based on bird's-eye view coordinates and output all targets in the area of ​​interest; The target detection types include motor vehicle targets and non-motor vehicle targets; According to the target detection type and box characteristics, the pixel coordinates of each detected target pixel are obtained in the pixel coordinate system and converted into the target coordinates as follows: When the target is a non-motor vehicle, there is a conversion formula: Where x and y are the target coordinates, x0, is the pixel coordinate, h is the height of the box; When the target belongs to a motor vehicle target and satisfies the relation When , there is a conversion formula: When the target belongs to a motor vehicle target and satisfies the relation When , a threshold is set for the aspect ratio of the box; When the aspect ratio of the target box is greater than the threshold, the conversion formula is: When the aspect ratio of the target box is less than the threshold, there is a conversion formula: In the above formula, x and y are the target coordinates, x0, is the pixel coordinate, h is the height of the box, w is the width of the box, and H is the height of a single frame image.

2. The method for multi-camera target fusion and blind spot filling according to claim 1, characterized in that: The specific formula for performing bird's-eye perspective transformation on the target coordinates in the region of interest to obtain the bird's-eye view coordinates is: Where u' and v' are bird's-eye view coordinates, k 11 、k 12 、k 13 、k 21 、k 22 、k 23 、k 31 、k 32 is the coefficient, and x and y are the target coordinates.

3. The method for multi-camera target fusion and blind spot filling according to claim 1, characterized in that: The specific method of performing target fusion based on bird's-eye view coordinates and outputting all targets in the region of interest is as follows: Set the distance threshold D min and D max , traverse the camera S with the largest number of targets in the area of ​​interest kmax and other cameras S i All goals; Traversing the Camera and Find all objects in the area of ​​interest and find the camera Each target in the ROI is The target with the closest Euclidean distance in the upper ROI, and record the closest distance value d min ; Compare d min With D min and D max The size of the value, if it satisfies the relationship d min >D max , it means that the two are not the same target; if the relationship d min <D min , it means that the two are the same goal and are integrated; if the relationship D is satisfied min <d min <D max , then determine whether the target types of the two are consistent. If so, they are the same target and are fused. If not, they are fused again in the next frame of video source image.

4. The method for multi-camera target fusion and blind spot filling according to claim 3, characterized in that: The specific method of performing target fusion based on bird's-eye view coordinates and outputting all targets in the region of interest also includes: Set the fusion threshold. If the fusion times exceed the fusion threshold and the fusion fails, the fusion will be stopped.

Images