Obstacle Fusion Method, System, Device and Medium under Multi-Camera Overlapping Field of View

Abstract

The present invention relates to the field of intelligent driving technology, and specifically relates to an obstacle fusion method, system, device and medium under the overlapping field of view of multiple cameras. First, obtain the original images collected by the cameras, calculate the measurement values of each obstacle target in the original images, match the obstacle targets. If the matching is successful, it means that the current obstacle target is making a cross-camera movement. According to the historical fusion results stored in the historical fusion list, compensate and correct the measurement values of the current obstacle target, perform data fusion on the compensated and corrected measurement values, and store them in the historical fusion list; if the matching fails, it means that the current obstacle target is not making a cross-camera movement, filter the measurement values of the current obstacle target, and store the filtered measurement values in the historical fusion list. By compensating and correcting the measurement values of the current obstacle target to reduce measurement errors, perform data fusion on the compensated and corrected measurement values, thereby improving the accuracy of the fusion results.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and particularly relates to an obstacle fusion method, system, device and medium under multi-camera overlapping fields of view. Background Art

[0002] An urgent problem to be solved in the obstacle information fusion technology under multi-camera overlapping fields of view is how to fuse the information of the same target sensed by multiple cameras, especially when some or all cameras cannot obtain complete information of the target (i.e., the target does not fully appear within the camera's field of view), into accurate and stable information.

[0003] In related technologies, it is proposed to install multiple different types of sensors on a vehicle, such as: visual sensors, radar sensors and laser sensors. In the target obstacle fusion stage, first obtain the target perception results of each sensor, and then project the obstacle information obtained by multiple sensors onto a 3D model of the vehicle's driving road according to the vehicle's driving position, delete the obstacle information projected alone, and fuse the remaining projections as the information to be fused in the 3D model. Finally, send this information to be fused into a judgment model for learning, and send the obstacle information determined by the judgment model to the vehicle's driving system as the final obstacle result.

[0004] This method uses multiple different types of sensors to obtain obstacle information. When multiple sensors can only obtain partial information of a certain target obstacle, that is, when the measurement differences between sensors are large, the method of filtering the fusion information by whether the projections coincide is no longer applicable, and it is difficult to completely fuse the complete information of the target obstacle. Since the judgment model is trained and learned using sample data, it is difficult to balance the system real-time performance and the fusion effect, and the use of multiple sensors results in higher hardware costs and system complexity.

[0005] In another related technology, a target obstacle tracking system based on fuzzy multi-sensor data fusion is proposed to overcome the limitations of using a single sensor. The data fusion process selects a linear fusion function of the sensor measurement to process, which can exclude invalid measurement in the estimation process to a certain extent and reduce the error of the fusion result.

[0006] This method lacks the prior statistical information of the sensors included in the target obstacle movement and estimation process. When there are certain measurement deviations in multiple sensors at a certain moment, this method will continuously accumulate and amplify the error of the fusion result, resulting in a deterioration of the fusion effect.

[0007] In another related technology, a data fusion method based on error distribution fitting is proposed. This method first obtains two sets of data to be fused, calculates the difference between these two sets of test sequences to obtain error data. Then, it performs distribution fitting on the error data, statistically analyzes the error data after distribution fitting to obtain the normal distribution parameters of the error, and superimposes the error data of the fitted normal distribution onto the data to be fused. Finally, a Kalman filter is used to obtain the fused data.

[0008] This method is applicable to situations with a large amount of observation data. In the obstacle information fusion under multiple cameras, the amount of obstacle information to be fused obtained at the same moment is small, and the error fluctuations of the obstacle information measured by each camera are large. If the validity of the original observation data is not judged and the calculated fitting error data and measurement results are directly superimposed, the accuracy of the fusion result will decrease.

[0009] In summary, existing fusion methods are all based on the fact that at least one sensor in the fused data can obtain complete information about the target, and then linear or non-linear fusion is performed on the original data. However, when all sensors can only obtain partial information about the target, or when the measured values of all sensors have large fluctuations, it is difficult for existing fusion schemes to ensure the accuracy and stability of the fusion result. Summary of the Invention

[0010] In view of this, the purpose of the present invention is to provide an obstacle fusion method, system, device and medium under the overlapping fields of view of multiple cameras, so as to solve the problem that it is difficult to ensure the accuracy and stability of the fusion result when the obstacle target information obtained by sensors is incomplete in the prior art.

[0011] According to the first aspect of the embodiments of the present invention, an obstacle fusion method under the overlapping fields of view of multiple cameras is provided, including:

[0012] Obtain the original images collected by the cameras;

[0013] Calculate the measured values of each obstacle target in the original images;

[0014] Match the obstacle targets under different cameras. If the match is successful, it is determined that the current obstacle target is making a cross-camera movement; if the match fails, it is determined that the current obstacle target is not making a cross-camera movement;

[0015] If it is determined that the current obstacle target is not making a cross-camera movement, filter the measured values of the current obstacle target and store the filtered measured values in the historical fusion list;

[0016] If it is determined that the current obstacle target is performing cross-camera movement, the measurement value of the current obstacle target is compensated and corrected according to the historical fusion results stored in the historical fusion list; for the compensated and corrected measurement value, data fusion is performed, and the fusion result is stored in the historical fusion list.

[0017] Preferably, calculating the measurement value of each obstacle target in the original image includes:

[0018] Detecting and identifying the obstacle targets in the original image to obtain the 3D box information of each obstacle target, where the 3D box information includes the grounding point of the obstacle target;

[0019] Calculating the grounding point coordinates of the grounding point of each obstacle target in the vehicle coordinate system, and the width, length dimensions and heading angle of each obstacle target;

[0020] Taking the grounding point coordinates of each obstacle target in the vehicle coordinate system, the width, length dimensions and heading angle of each obstacle target as the measurement value of each obstacle target.

[0021] Preferably, compensating and correcting the measurement value of the current obstacle target includes:

[0022] Obtaining the historical fusion result of the current obstacle target, where the historical fusion result at least includes: the tracking ID of the obstacle target, the width, length dimensions, the data distribution parameters that the width, length dimensions obey;

[0023] Calculating the positioning confidence of each grounding point of the current obstacle target;

[0024] Compensating and correcting the width, length dimensions, heading angle and the grounding point coordinates in the vehicle coordinate system of the current obstacle target in sequence according to the positioning confidence and the historical fusion result.

[0025] Preferably, the method further includes:

[0026] Calculating the center point coordinates of the current obstacle target according to the compensated and corrected grounding point coordinates in the vehicle coordinate system;

[0027] Performing Kalman filtering on the center point coordinates and the compensated and corrected heading angle of the current obstacle target, and referring to the measurement noise to obtain the filtered center point coordinates and heading angle of the current obstacle target;

[0028] Then, performing data fusion on the corrected measurement value specifically means:

[0029] Performing data fusion on the filtered center point coordinates and heading angle of the current obstacle target.

[0030] Preferably, the central point coordinates of the current obstacle target and the compensated and corrected heading angle are subjected to Kalman filtering, and with reference to the measurement noise, the filtered central point coordinates and heading angle of the current obstacle target are obtained, including:

[0031] Input the central point coordinates of the current obstacle target and the compensated and corrected heading angle into the first Kalman filter, so that the first Kalman filter predicts the position of the current obstacle target at the current moment through the CTRV motion model, and obtains the predicted central point coordinates and heading angle; and,

[0032] Enable the first Kalman filter to update the predicted central point coordinates and heading angle according to the measurement noise fed back by the current fusion result, and output the filtered central point coordinates and heading angle.

[0033] Preferably, the data fusion of the compensated and corrected measurement values includes:

[0034] Judge the historical matching times of two obstacle targets to be fused. If the historical matching times are greater than or equal to the threshold, input the central point coordinates and heading angle of the two obstacle targets to be fused into the second Kalman filter for data fusion, and input the width and length dimensions of the two obstacle targets to be fused into the dimension filter for data fusion;

[0035] If the historical matching times are less than the threshold, fuse the grounding point coordinates of the two obstacle targets to be fused, recalculate the central point coordinates and heading angle of the current obstacle target according to the fused grounding point coordinates, input the recalculated central point coordinates and heading angle into the second Kalman filter for data fusion, and input the recalculated width and length dimensions into the dimension filter for data fusion.

[0036] Preferably, the fusion of the grounding point coordinates of the two obstacle targets to be fused includes:

[0037] For the two grounding points in the length direction of the two obstacle targets to be fused, respectively select the grounding point coordinates with high positioning confidence as the fused first grounding point coordinates and second grounding point coordinates.

[0038] Preferably, the recalculation of the central point coordinates and heading angle of the current obstacle target according to the fused grounding point coordinates includes:

[0039] Recalculate the length and heading angle of the current obstacle target according to the fused first grounding point coordinates and second grounding point coordinates;

[0040] For the widths of the two obstacle targets to be fused, select the width corresponding to the grounding point with high positioning confidence as the fused width;

[0041] Calculate the ground contact point coordinates in the width direction of the current obstacle target based on the fused second ground contact point coordinates, width, and recalculated heading angle, and use them as the fused third ground contact point coordinates;

[0042] Recalculate the center point coordinates of the current obstacle target based on the fused first ground contact point coordinates and the third ground contact point coordinates.

[0043] Preferably, the method further includes:

[0044] Take the absolute value of the difference between the center point coordinates before and after data fusion as the measurement error of the center point coordinates;

[0045] Take the absolute value of the difference between the heading angles before and after data fusion as the measurement error of the heading angle.

[0046] According to the second aspect of the embodiments of the present invention, there is provided an obstacle fusion system under a multi-camera overlapping field of view, including:

[0047] An acquisition module, configured to acquire the original images collected by the cameras;

[0048] A calculation module, configured to calculate the measurement values of each obstacle target in the original images;

[0049] A matching module, configured to match the obstacle targets under different cameras. If the matching is successful, it is determined that the current obstacle target is making a cross-camera movement; if the matching fails, it is determined that the current obstacle target is not making a cross-camera movement;

[0050] A processing module, configured to, if it is determined that the current obstacle target is not making a cross-camera movement, filter the measurement values of the current obstacle target and store the filtered measurement values in the historical fusion list;

[0051] It is also configured to, if it is determined that the current obstacle target is making a cross-camera movement, compensate and correct the measurement values of the current obstacle target according to the historical fusion results stored in the historical fusion list; perform data fusion on the compensated and corrected measurement values, and store the fusion results in the historical fusion list.

[0052] According to the third aspect of the embodiments of the present invention, there is provided an electronic device, including:

[0053] A communication module, a processor, and a memory, wherein program instructions are stored in the memory;

[0054] The processor is configured to execute the program instructions stored in the memory to execute the above method.

[0055] According to the fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, on which a rewritable computer program is stored;

[0056] When the computer program runs on a computer device, the computer device is caused to execute the above-mentioned method.

[0057] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0058] When it is determined that the current obstacle target is performing cross-camera movement, according to the historical fusion results stored in the historical fusion list, the measurement value of the current obstacle target is compensated and corrected to reduce the measurement error, and data fusion is performed on the compensated and corrected measurement value, thereby improving the accuracy of the fusion result; since the data fusion fuses the measurement values of the obstacle targets compensated and corrected under different cameras, the stability of the fusion result is improved, and further solves the problem that it is difficult to ensure the accuracy and stability of the fusion result in the prior art when the obstacle target information obtained by the sensor is incomplete.

[0059] In addition, the technical solution provided by the present invention does not rely on other types of sensors except for cameras, so it can greatly reduce the complexity of the system and the hardware cost.

[0060] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.

[0062] Figure 1 is a flowchart of a method for fusing obstacles in a multi-camera overlapping field of view shown according to an exemplary embodiment;

[0063] Figure 2 is a schematic diagram of a 3D box of an obstacle target shown according to an exemplary embodiment;

[0064] Figure 3 is a schematic diagram of calculating the measurement value of an obstacle target shown according to an exemplary embodiment;

[0065] Figure 4 is a schematic diagram of an obstacle target performing cross-camera movement shown according to an exemplary embodiment;

[0066] Figure 5 is a flowchart of data fusion shown according to an exemplary embodiment;

[0067] Figure 6 is a schematic diagram of two obstacle targets to be fused in a vehicle coordinate system shown according to an exemplary embodiment;

[0068] Figures 7A - 7B It is a schematic diagram of two obstacle targets to be fused in the original image shown according to an exemplary embodiment;

[0069] Figure 8 It is a schematic block diagram of an obstacle fusion system under the overlapping field of view of multiple cameras shown according to an exemplary embodiment;

[0070] Figure 9 It is a data processing flow chart inside the processing module shown according to an exemplary embodiment. Specific Embodiments

[0071] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0072] It should be noted that the "current vehicle" mentioned in each embodiment of the present invention refers to the vehicle itself that can run the method and system for obstacle fusion under the overlapping field of view of multiple cameras provided by the present invention.

[0073] The "obstacle target" mentioned in each embodiment of the present invention includes various static obstacle targets and / or dynamic obstacle targets, especially refers to obstacle vehicles. The "obstacle vehicle" refers to a vehicle other than the vehicle itself that may affect the driving of the vehicle itself.

[0074] Embodiment 1

[0075] Please refer to Figure 1 , Figure 1 It is a flow chart of a method for obstacle fusion under the overlapping field of view of multiple cameras shown according to an exemplary embodiment. As Figure 1 shown, the method includes:

[0076] Step S11: Obtain the original image collected by the camera;

[0077] Step S12: Calculate the measurement value of each obstacle target in the original image;

[0078] Step S13: Match the obstacle targets under different cameras. If the match is successful, it is determined that the current obstacle target is making a cross-camera movement; if the match fails, it is determined that the current obstacle target is not making a cross-camera movement;

[0079] Step S14: If it is determined that the current obstacle target is not making a cross-camera movement, filter the measurement value of the current obstacle target and store the filtered measurement value in the historical fusion list;

[0080] Step S15: If it is determined that the current obstacle target is performing cross-camera movement, compensate and correct the measurement value of the current obstacle target according to the historical fusion results stored in the historical fusion list; perform data fusion on the compensated and corrected measurement value, and store the fusion result in the historical fusion list.

[0081] It should be noted that the application scenarios applicable to the technical solution provided in this embodiment include, but are not limited to: autonomous driving, assisted driving, etc. of vehicles. When the technical solution provided in this embodiment is actually used, it can be loaded in the central control system of the current vehicle for use, or can be loaded in an electronic device for use; the electronic device includes, but is not limited to: in-vehicle computers and external computer devices.

[0082] Subsequently, based on the solution provided in this embodiment, solutions such as voice reminder, text reminder, graphic reminder (including reminder with different icon colors and shapes), animation reminder, etc. for obstacle targets in the overlapping field of view of multiple cameras of the current vehicle are all within the protection scope of this embodiment.

[0083] It can be understood that for the technical solution provided in this embodiment, when it is determined that the current obstacle target is performing cross-camera movement, the measurement value of the current obstacle target is compensated and corrected according to the historical fusion results stored in the historical fusion list to reduce measurement errors, and data fusion is performed on the compensated and corrected measurement value, thereby improving the accuracy of the fusion result; since the data fusion is the fusion of the compensated and corrected measurement values of the obstacle targets under different cameras, the stability of the fusion result is improved, and thus the problem in the prior art that it is difficult to ensure the accuracy and stability of the fusion result when the obstacle target information obtained by the sensor is incomplete is solved.

[0084] In addition, the technical solution provided in this embodiment does not rely on other types of sensors except for cameras, so it can greatly reduce the complexity of the system and the hardware cost.

[0085] In specific practice, for "calculating the measurement value of each obstacle target in the original image" in step S12, there can be multiple implementation manners, and one implementation manner can be:

[0086] 1. Detect and identify the obstacle targets in the original image (for example, detect and identify the obstacles in the image through neural network models such as YOLO and ResNet) to obtain the 3D box information of each obstacle target, and the 3D box information includes the grounding point of the obstacle target (as Figure 2 shown, the 3D box of each obstacle target contains 6 image coordinate points P i(x, y), where i = 1, 2, 3, 4, 5, 6, and P1, P2, P3 represent the grounding points of the obstacle targets);

[0087] 2. Calculate the grounding point coordinates of each obstacle target in the vehicle coordinate system, as well as the width-length dimensions and heading angle of each obstacle target;

[0088] Taking the center of the current vehicle as the coordinate origin, the X-axis is parallel to the ground and points in the direction of the current vehicle's travel, the Y-axis is parallel to the ground and points to the left of the driver, and the Z-axis is perpendicular to the ground and points in the height direction of the current vehicle, establish a vehicle coordinate system, and according to the camera projection model formula:

[0089]

[0090] where M1 is the camera internal parameter matrix and M2 is the camera external parameter matrix, the grounding point coordinates P j (X, Y, 0) (j = 1, 2, 3) of each obstacle target in the vehicle coordinate system can be calculated. At the same time, based on the grounding point coordinates P j (X, Y, 0), calculate the width-length dimensions [w, l] and heading angle θ of each obstacle target. θ is defined as the angle between the forward direction of the obstacle target and the positive direction of the X-axis of the vehicle coordinate system, and the value range is [0, 2π].

[0091] See Figure 3 , the calculation formulas for the width-length dimensions [w, l] and heading angle θ of the obstacle target are as follows:

[0092] w = |P2P3| (1)

[0093] l = |P2P1| (2)

[0094]

[0095] 3. Take the grounding point coordinates P j (X, Y, 0), width-length dimensions [w, l], and heading angle θ of each obstacle target in the vehicle coordinate system as the measurement values of each obstacle target.

[0096] In specific practice, there can be multiple implementation methods for "compensating and correcting the measurement values of the current obstacle target" in step S15. One of the implementation methods can be:

[0097] 1. Obtain the historical fusion result of the current obstacle target. The historical fusion result at least includes: the tracking ID of the obstacle target, width-length dimensions [w h , l h , heading angle θ h , the data distribution that the width-length dimensions follow

[0098] Cloth parameters (assuming that the mean of the obstacle target size after N - time historical fusion is μ and the variance is σ, and assuming that the size measurement value of the obstacle target follows a Gaussian distribution, i.e., w ~ N(μ w , σ w ), l ~ N(μ l , σ l ))。

[0099] 2. Calculate the positioning confidence of each grounding point of the current obstacle target.

[0100] Refer to Figure 4 . When an obstacle target is moving across cameras, the obstacle target cannot appear completely in the original image captured by the camera, resulting in a decrease in the confidence of the grounding point coordinates of the 3D box and an increase in the calculation error. At this time, it is necessary to evaluate the positioning confidence C j of the grounding point P j . If the current obstacle target has no historical fusion information in the historical fusion list, it means that the current obstacle target appears in the field of view of the current vehicle for the first time. Directly set the confidence of the non - out - of - bounds point to 1 and the out - of - bounds point to 0. If the current obstacle target has historical fusion information in the historical fusion list, assume that the size information of the current obstacle target in the historical fusion list is [w h , l h , and the heading angle at the previous moment is θ h . Then the positioning confidence of each grounding point of the current obstacle target is:

[0101]

[0102] Among them, w and l respectively represent the width and length dimensions of the obstacle target at the current moment, and θ represents the heading angle of the obstacle target at the current moment. This calculation formula is designed according to the logical relationship between each grounding point P j of the obstacle and the size and heading angle θ.

[0103] 3. If the current obstacle target has historical fusion information in the historical fusion list, first, correct the width and length dimensions of the obstacle target. Assume that the corrected width and length dimensions are [w c , l c . Then

[0104]

[0105]

[0106]

[0107] Among them, CI represents the confidence interval threshold when the confidence level is α, which can be obtained through the normal distribution lookup table.

[0108] Secondly, let the corrected heading angle be θ c , then:

[0109] θ c = C2·θ+(1 - C2)·θ h

[0110] where C2 represents the fixed - position confidence of the grounding point P2.

[0111] Finally, through the corrected width - length dimensions [w c , l c , the heading angle θ c and the point P1, correct P2 and P3 through the above formulas (1)-(3) to obtain the compensated and corrected P c2 and P c3 .

[0112] According to the compensated and corrected grounding - point coordinates in the vehicle coordinate system, calculate the center - point coordinates P c =(P1 + P c3 ) / 2;

[0113] Input the center - point coordinates P c of the current obstacle target and the compensated and corrected heading angle θ c into the first Kalman filter, so that the first Kalman filter predicts the position of the current obstacle target at the current moment through the CTRV motion model to obtain the predicted center - point coordinates and the heading angle and, enable the first Kalman filter to update the predicted center - point coordinates and the heading angle according to the measurement noise fed back by the current fusion result, and output the filtered center - point coordinates P' c and the heading angle θ' c .

[0114] Among them, the first Kalman filter predicts the position of the current obstacle target at the current moment through the CTRV motion model to obtain the predicted center - point coordinates and the heading angle Specifically:

[0115] Let the state space of the first Kalman filter be:

[0116]

[0117] where P x , P y respectively represent the x - coordinate and y - coordinate of the center point , v represents the velocity vector, and θ represents the target heading angle Let φ represent the angular velocity, then the CTRV prediction equation is as follows:

[0118]

[0119] In specific practice, there are various implementation manners for "performing data fusion on the compensated and corrected measurement values" in step S15. One implementation manner can be:

[0120] Please refer to Figure 5 , Figure 5 which is the data fusion flowchart:

[0121] 1) Determine the historical matching times of two obstacle targets to be fused. If the historical matching times ≥ N (N is a threshold, N is a positive integer, and the specific value is set according to historical experience values or experimental data), then the center point coordinates P′ c and the heading angle θ′ c of the two obstacle targets to be fused are passed into the second Kalman filter for data fusion. At the same time, the width and length dimensions [w c , l c of the two obstacle targets to be fused are passed into the dimension filter for data fusion.

[0122] It can be understood that since the size of the obstacle target is fixed and does not change with time, the dimension filter can select median filtering or mean filtering to obtain stable and accurate results. In this embodiment, no specific limitation is made.

[0123] 2) If the historical matching times < N, then the grounding point coordinates of the two obstacle targets to be fused are fused (for example, the grounding point coordinates P Figure 6 in 1j (X 1j , Y 1j , 0) and P 2j (X 2j , Y 2j , 0), where j = 1, 2, 3, are fused respectively), and the center point coordinates and heading angle of the current obstacle target are recalculated according to the fused grounding point coordinates. The recalculated center point coordinates and heading angle are input into the second Kalman filter for data fusion, and the recalculated width and length dimensions are input into the dimension filter for data fusion.

[0124] See Figure 7A and Figure 7B . Suppose the two obstacle targets to be fused are the vehicle in Figure 7A and the vehicle in Figure 7B . The fusion of the grounding point coordinates of the two obstacle targets to be fused includes:

[0125] For the two grounding points in the length direction of the two obstacle targets to be fused, the coordinates of the grounding points with high positioning confidence are respectively selected as the coordinates of the first grounding point P1 and the second grounding point P2 after fusion.

[0126] Let the coordinates of the grounding point after fusion be P j (X, Y, 0) (j = 1, 2, 3), P 1j (X 1j , Y 1j , 0) and P 2j (X 2j , Y 2j , 0), where j = 1, 2, 3, and the corresponding positioning confidences are C 1j and C 2j , then

[0127]

[0128]

[0129] According to the coordinates of the first grounding point P1 and the second grounding point P2 after fusion, recalculate the length l and the heading angle θ of the current obstacle target, including:

[0130] Length of the obstacle target:

[0131] l = |P1P2|

[0132] Heading angle:

[0133]

[0134] For the widths of the two obstacle targets to be fused, select the width corresponding to the grounding point with high positioning confidence as the width after fusion, specifically:

[0135]

[0136] According to the coordinates of the second grounding point P2, the width, and the recalculated heading angle after fusion, calculate the coordinates of the grounding point in the width direction of the current obstacle target as the coordinates of the third grounding point P3 after fusion, specifically:

[0137]

[0138] According to the coordinates of the first grounding point P1 and the third grounding point P3 after fusion, recalculate the center point coordinates P c , specifically:

[0139] P c = (P1 + P3) / 2

[0140] According to the fusion result, the measurement noise can be obtained, including:

[0141] 1. Take the absolute value of the difference between the center point coordinates before and after data fusion as the measurement error of the center point coordinates:

[0142] Among them, is the center point coordinate after data fusion, and P c is the center point coordinate before data fusion.

[0143] 2. Take the absolute value of the difference between the heading angles before and after data fusion as the measurement error of the heading angle:

[0144] Among them, is the heading angle after data fusion, and θ is the heading angle before data fusion.

[0145] It can be understood that the technical solution provided in this embodiment, when determining that the current obstacle target is performing cross-camera movement, compensates and corrects the measurement value of the current obstacle target according to the historical fusion results stored in the historical fusion list to reduce the measurement error, and proposes an evaluation method for the confidence of obstacle target measurement information, laying a foundation for improving the accuracy of the fusion result;

[0146] In addition, the technical solution provided in this embodiment proposes a joint fusion method for obstacle size and pose information (including the center point coordinates and heading angle of the obstacle target), which can improve the fusion accuracy of size and pose at the same time, and has good user experience and high satisfaction.

[0147] Embodiment 2

[0148] Please refer to Figure 8 , Figure 8 which is a schematic block diagram of an obstacle fusion system in a multi-camera overlapping field of view shown in an exemplary embodiment. As Figure 8 shown, the obstacle fusion system 800 in the multi-camera overlapping field of view includes:

[0149] An acquisition module 801, configured to acquire the original images collected by the cameras;

[0150] A calculation module 802, configured to calculate the measurement values of each obstacle target in the original images;

[0151] A matching module 803, configured to match the obstacle targets under different cameras. If the match is successful, it is determined that the current obstacle target is performing cross-camera movement; if the match fails, it is determined that the current obstacle target is not performing cross-camera movement;

[0152] A processing module 804, configured to filter the measurement value of the current obstacle target if it is determined that the current obstacle target does not perform cross-camera movement, and store the filtered measurement value in the historical fusion list;

[0153] It is also configured to, if it is determined that the current obstacle target is performing cross-camera movement, compensate and correct the measurement value of the current obstacle target according to the historical fusion result stored in the historical fusion list; perform data fusion on the compensated and corrected measurement value, and store the fusion result in the historical fusion list.

[0154] It should be noted that, in specific practice, the internal processing flow of the processing module 804 can be referred to Figure 9 as shown. Refer to Figure 9 , and the processing module 804 specifically includes:

[0155] A first filter, configured to filter the measurement value of the current obstacle target if it is determined that the current obstacle target does not perform cross-camera movement, and store the filtered measurement value in the historical fusion list.

[0156] An information correction module, configured to obtain the historical fusion result of the current obstacle target if it is determined that the current obstacle target is performing cross-camera movement, and the historical fusion result at least includes: the tracking ID of the obstacle target, the width and length dimensions, the heading angle, and the data distribution parameters that the width and length dimensions follow;

[0157] Calculate the fixed-position confidence of each grounding point of the current obstacle target;

[0158] Compensate and correct the width and length dimensions, the heading angle, and the grounding point coordinates in the vehicle coordinate system of the current obstacle target in sequence according to the fixed-position confidence and the historical fusion result;

[0159] Calculate the center point coordinates of the current obstacle target according to the compensated and corrected grounding point coordinates in the vehicle coordinate system.

[0160] A second filter, which can specifically be a first Kalman filter, configured to perform Kalman filtering on the center point coordinates and the compensated and corrected heading angle of the current obstacle target, and refer to the measurement noise fed back by the data fusion module to obtain the filtered center point coordinates and heading angle of the current obstacle target.

[0161] A data fusion module, which is internally provided with a second Kalman filter and a size filter, configured to perform data fusion on the compensated and corrected measurement value, and store the fusion result in the historical fusion list.

[0162] It should be noted that for the implementation manners and beneficial effects of each module in this embodiment, reference can be made to the introduction of the relevant steps in Embodiment 1, and details will not be described in this embodiment.

[0163] It should be noted that the application scenarios applicable to the technical solution provided in this embodiment include, but are not limited to: autonomous driving, assisted driving, etc. of vehicles. When the technical solution provided in this embodiment is actually used, it can be loaded and used in the central control system of the current vehicle, or can be loaded and used in an electronic device; the electronic device includes, but is not limited to: in-vehicle computers and external computer devices.

[0164] Subsequently, based on the solution provided in this embodiment, solutions such as voice reminders, text reminders, graphic reminders (including reminders using different icon colors and shapes), and animation reminders for the obstacle targets recognized by the current vehicle are all within the protection scope of this embodiment.

[0165] It can be understood that for the technical solution provided in this embodiment, when it is determined that the current obstacle target is performing cross-camera movement, based on the historical fusion results stored in the historical fusion list, the measured values of the current obstacle target are compensated and corrected to reduce measurement errors, and the compensated and corrected measured values are subjected to data fusion, thereby improving the accuracy of the fusion results; since the data fusion is the fusion of the compensated and corrected measured values of the obstacle targets under different cameras, the stability of the fusion results is improved, and further, the problem that it is difficult to ensure the accuracy and stability of the fusion results when the obstacle target information obtained by the sensor is incomplete in the prior art is solved.

[0166] In addition, the technical solution provided in this embodiment does not rely on other types of sensors except for cameras, so the complexity of the system and the hardware cost can be greatly reduced.

[0167] Embodiment Three

[0168] An electronic device shown according to an exemplary embodiment includes:

[0169] A communication module, a processor, and a memory, wherein program instructions are stored in the memory;

[0170] The processor is used to execute the program instructions stored in the memory to execute the above-mentioned obstacle fusion method under the overlapping fields of view of multiple cameras.

[0171] It should be noted that the electronic device includes, but is not limited to: in-vehicle computers and external computer devices. The communication module includes, but is not limited to: wired communication modules and wireless communication modules, such as: WCDMA, GSM, CDMA, and / or LTE communication modules, ZigBee modules, Bluetooth modules, Wi-Fi modules, etc.

[0172] The processor includes, but is not limited to: CPU, single-chip microcomputer, PLC controller, FPGA controller, etc.

[0173] The memory may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory; it may also include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0174] It can be understood that for the technical solution provided in this embodiment, when it is determined that the current obstacle target is performing cross-camera movement, the measured value of the current obstacle target is compensated and corrected according to the historical fusion results stored in the historical fusion list to reduce measurement errors, and data fusion is performed on the compensated and corrected measured value, thereby improving the accuracy of the fusion result; since the data fusion fuses the measured values of the obstacle targets under different cameras after compensation and correction, the stability of the fusion result is improved, and further solves the problem in the prior art that it is difficult to ensure the accuracy and stability of the fusion result when the obstacle target information obtained by the sensor is incomplete.

[0175] In addition, the technical solution provided in this embodiment does not rely on other types of sensors except for cameras, so it can greatly reduce the complexity of the system and the hardware cost.

[0176] Embodiment Four

[0177] A computer-readable storage medium according to an exemplary embodiment stores a rewritable computer program;

[0178] When the computer program runs on a computer device, the computer device is caused to execute the above-mentioned obstacle fusion method in the overlapping field of view of multiple cameras.

[0179] The computer-readable storage medium disclosed in this embodiment includes, but is not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or component.

[0180] It can be understood that for the technical solution provided in this embodiment, when it is determined that the current obstacle target is performing cross-camera movement, the measured value of the current obstacle target is compensated and corrected according to the historical fusion results stored in the historical fusion list to reduce measurement errors, and data fusion is performed on the compensated and corrected measured value, thereby improving the accuracy of the fusion result; since the data fusion fuses the measured values of the obstacle targets under different cameras after compensation and correction, the stability of the fusion result is improved, and further solves the problem in the prior art that it is difficult to ensure the accuracy and stability of the fusion result when the obstacle target information obtained by the sensor is incomplete.

[0181] In addition, the technical solution provided in this embodiment does not rely on other types of sensors except for cameras, so it can greatly reduce the complexity of the system and the hardware cost.

[0182] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be seen in the same or similar content in other embodiments.

[0183] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field of the embodiments of the present invention.

[0184] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0185] Those of ordinary skill in the technical field of the present invention can understand that all or part of the steps carried by the methods in the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0186] In addition, each functional unit in various embodiments of the present invention may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0187] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, or the like.

[0188] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0189] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for fusing obstacles in the overlapping fields of view of multiple cameras, characterized in that, Including: Obtaining the original image collected by the camera; Calculating the measurement value of each obstacle target in the original image, including: detecting and identifying the obstacle targets in the original image to obtain the 3D bounding box information of each obstacle target, where the 3D bounding box information includes the grounding point of the obstacle target; calculating the grounding point coordinates of the grounding point of each obstacle target in the vehicle coordinate system, as well as the width, length dimensions and heading angle of each obstacle target; using the grounding point coordinates of each obstacle target in the vehicle coordinate system, the width, length dimensions and heading angle of each obstacle target as the measurement value of each obstacle target; Matching the obstacle targets under different cameras. If the matching is successful, it is determined that the current obstacle target is performing cross-camera movement; if the matching fails, it is determined that the current obstacle target is not performing cross-camera movement; If it is determined that the current obstacle target is not performing cross-camera movement, filtering the measurement value of the current obstacle target and storing the filtered measurement value in the historical fusion list; If it is determined that the current obstacle target is performing cross-camera movement, compensating and correcting the measurement value of the current obstacle target according to the historical fusion result stored in the historical fusion list; performing data fusion on the compensated and corrected measurement value and storing the fusion result in the historical fusion list; The compensating and correcting the measurement value of the current obstacle target includes: Obtaining the historical fusion result of the current obstacle target, where the historical fusion result at least includes: the tracking ID of the obstacle target, the width, length dimensions, the data distribution parameters that the width, length dimensions follow; Calculating the positioning confidence of each grounding point of the current obstacle target, and the calculation method is: Among them, C j is the positioning confidence, and w and l respectively represent the width and length dimensions of the obstacle target at the current moment. represents the heading angle of the obstacle target at the current moment. This calculation formula is based on the grounding point P of each obstacle j and the dimensions and heading angle designed according to the logical relationship; is the heading angle at the previous moment; w h It is the width in the size information of the current obstacle target in the historical fusion list; l h is the length in the size information of the current obstacle target in the historical fusion list; Compensating and correcting the width, length dimensions, heading angle and the grounding point coordinates in the vehicle coordinate system of the current obstacle target in sequence according to the positioning confidence and the historical fusion result.

2. The method according to claim 1, characterized in that, Also including: Calculating the center point coordinates of the current obstacle target according to the compensated and corrected grounding point coordinates in the vehicle coordinate system; Performing Kalman filtering on the center point coordinates and the compensated and corrected heading angle of the current obstacle target, and referring to the measurement noise to obtain the filtered center point coordinates and heading angle of the current obstacle target; Then, the performing data fusion on the compensated and corrected measurement value is specifically: Performing data fusion on the filtered center point coordinates and heading angle of the current obstacle target.

3. The method according to claim 2, wherein The performing Kalman filtering on the center point coordinates and the compensated and corrected heading angle of the current obstacle target, and referring to the measurement noise to obtain the filtered center point coordinates and heading angle of the current obstacle target includes: Inputting the center point coordinates and the compensated and corrected heading angle of the current obstacle target into the first Kalman filter, so that the first Kalman filter predicts the position of the current obstacle target at the current moment through the CTRV motion model to obtain the predicted center point coordinates and heading angle; and, Making the first Kalman filter update the predicted center point coordinates and heading angle according to the measurement noise fed back by the current fusion result and output the filtered center point coordinates and heading angle.

4. The method according to claim 3, wherein The performing data fusion on the compensated and corrected measurement value includes: Judge the historical matching times of two obstacle targets to be fused. If the historical matching times are greater than or equal to the threshold, input the center point coordinates and heading angles of the two obstacle targets to be fused into the second Kalman filter for data fusion, and input the width and length dimensions of the two obstacle targets to be fused into the dimension filter for data fusion; If the historical matching times are less than the threshold, fuse the grounding point coordinates of the two obstacle targets to be fused, recalculate the center point coordinates and heading angles of the current obstacle target according to the fused grounding point coordinates, input the recalculated center point coordinates and heading angles into the second Kalman filter for data fusion, and input the recalculated width and length dimensions into the dimension filter for data fusion.

5. The method according to claim 4, characterized in that The fusing of the grounding point coordinates of the two obstacle targets to be fused includes: For the two grounding points in the length direction of the two obstacle targets to be fused, respectively select the grounding point coordinates with high positioning confidence as the first and second grounding point coordinates after fusion.

6. The method according to claim 5, characterized in that, The recalculating of the center point coordinates and heading angles of the current obstacle target according to the fused grounding point coordinates includes: Recalculate the length and heading angle of the current obstacle target according to the fused first and second grounding point coordinates; For the widths of the two obstacle targets to be fused, select the width corresponding to the grounding point with high positioning confidence as the width after fusion; Calculate the grounding point coordinates in the width direction of the current obstacle target according to the fused second grounding point coordinates, width and recalculated heading angle as the third grounding point coordinates after fusion; Recalculate the center point coordinates of the current obstacle target according to the fused first and third grounding point coordinates.

7. The method according to any one of claims 4 to 6, characterized in that, It also includes: Take the absolute value of the difference between the center point coordinates before and after data fusion as the measurement error of the center point coordinates; Take the absolute value of the difference between the heading angles before and after data fusion as the measurement error of the heading angles.

8. An obstacle fusion system under the overlapping fields of view of multiple cameras, characterized in that, It includes: An acquisition module for acquiring the original images collected by the camera; A calculation module for calculating the measurement values of each obstacle target in the original image, specifically for detecting and identifying the obstacle targets in the original image to obtain the 3D box information of each obstacle target, and the 3D box information includes the grounding points of the obstacle targets; Calculate the grounding point coordinates of the grounding points of each obstacle target in the vehicle coordinate system, as well as the width, length dimensions and heading angles of each obstacle target; take the grounding point coordinates of each obstacle target in the vehicle coordinate system, the width, length dimensions and heading angles of each obstacle target as the measurement values of each obstacle target; A matching module for matching the obstacle targets under different cameras. If the matching is successful, it is determined that the current obstacle target is making a cross-camera movement; If the matching fails, it is determined that the current obstacle target has not made a cross-camera movement; A processing module for, if it is determined that the current obstacle target has not made a cross-camera movement, filtering the measurement values of the current obstacle target and storing the filtered measurement values in the historical fusion list; It is also used to, if it is determined that the current obstacle target is performing cross-camera movement, compensate and correct the measurement value of the current obstacle target according to the historical fusion results stored in the historical fusion list; perform data fusion on the compensated and corrected measurement value, and store the fusion result in the historical fusion list; Specifically, it is used to obtain the historical fusion results of the current obstacle target, and the historical fusion results at least include: the tracking ID of the obstacle target, the width and length dimensions, the heading angle, and the data distribution parameters that the width and length dimensions follow; Calculate the positioning confidence of each grounding point of the current obstacle target, and the calculation method is: Among them, C j is the positioning confidence, and w and l respectively represent the width and length dimensions of the obstacle target at the current moment. represents the heading angle of the obstacle target at the current moment. The calculation formula is based on the grounding point P of each obstacle j and the dimensions and heading angle designed according to the logical relationship; is the heading angle at the previous moment; w h is the width in the size information of the current obstacle target in the historical fusion list; l h is the length in the size information of the current obstacle target in the historical fusion list; According to the positioning confidence and the historical fusion results, sequentially compensate and correct the width and length dimensions, the heading angle, and the grounding point coordinates in the vehicle coordinate system of the current obstacle target.

9. An electronic device, characterized in that, It includes: A communication module, a processor, and a memory, wherein program instructions are stored in the memory; The processor is used to execute the program instructions stored in the memory and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a rewritable computer program thereon; When the computer program runs on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 7.

Images