A method, device, equipment and medium for sensing target objects

Abstract

The present application discloses a method, device, equipment and medium for object perception, which relates to the field of object detection, including: using a two-dimensional perception model and a three-dimensional perception model to respectively detect an object of interest on a current acquired image of a vehicle-mounted monocular camera, so as to obtain corresponding first perception information of each first object and second perception information of each second object; when the first detection distance of the three-dimensional perception model is less than the second detection distance of the two-dimensional perception model, determining a first object to be fused within the first detection distance from each first object, and determining a second object to be fused belonging to the same object from each second object, and fusing the perception information corresponding to the first object to be fused and the second object to be fused to obtain the final perception information. The present application improves the perception accuracy of the object while ensuring the detection accuracy by fusing the object perception information from multiple sources within a smaller detection distance.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a target object sensing method, device, equipment and medium. Background Art

[0002] At present, intelligent front-view cameras are widely used in the field of driving assistance, and can provide end users with driving assistance functions such as adaptive cruise control, active braking, lane keeping, etc. Among them, for the perception scheme of the front-view camera, one is to perform two-dimensional object recognition between image layers, and then convert it to three-dimensional world space for corresponding post-processing to obtain the three-dimensional position information of the target object; the other is to directly perform three-dimensional object recognition between image layers to directly predict the three-dimensional position information of the target object.

[0003] However, for the above two perception schemes, the three-dimensional position information of the target object comes from a single perception mode, lacking perception redundancy. That is, when the perception mode has a false detection, there is no redundant information to avoid these false detections, which ultimately causes abnormal performance of the driving assistance function. In addition, the current perception schemes mostly use a combination of multiple sensors such as cameras and radars to realize object recognition between image layers, which requires relatively high hardware costs. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for object perception, which can fuse the object perception information from multiple sources within a small detection distance when using only a vehicle-mounted monocular camera, thereby improving the accuracy of object perception while ensuring the accuracy of object detection, and reducing the cost required for object perception. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a target object perception method, comprising:

[0006] Performing target object detection on a current acquired image of the vehicle-mounted monocular camera using a two-dimensional perception model to obtain first perception information of each first target object, and performing target object detection on the current acquired image using a three-dimensional perception model to obtain second perception information of each second target object;

[0007] When the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, determining a first target object to be fused that meets a first preset condition from each of the first targets; the first preset condition is that the distance of the target object relative to the vehicle in the direction of the front of the vehicle is not greater than the first detection distance;

[0008] Based on the first target to be fused, a corresponding second target to be fused is determined from each of the second targets, and final perception information is obtained by fusing the perception information corresponding to the first target to be fused and the second target to be fused respectively; the second target to be fused and the first target to be fused belong to the same target.

[0009] Optionally, the using of the two-dimensional perception model to perform target object detection on the current acquired image of the vehicle-mounted monocular camera to obtain first perception information of each first target object includes:

[0010] Detecting an object in the currently acquired image using a two-dimensional perception model to obtain a current moment detection frame for each first object, and performing association matching on each current moment detection frame and each current moment tracking frame to obtain a first association matching result; the current moment tracking frame is a tracking frame obtained by predicting a tracking frame of the object at a previous moment based on the two-dimensional perception model; wherein the tracking frame of the object at the first moment by the two-dimensional perception model is a tracking frame determined based on the detection frame of the object at the first moment by the two-dimensional perception model;

[0011] The corresponding current moment tracking frame is updated based on the current moment detection frame using the first association matching result, and the first perception information of the corresponding first target object is determined based on the updated current moment tracking frame.

[0012] Optionally, the performing association matching on each current moment detection frame and each current moment tracking frame to obtain a first association matching result includes:

[0013] Determine a first cost matrix based on the first generation value between each current moment detection frame and each current moment tracking frame, and perform association matching on each current moment detection frame and each current moment tracking frame based on the first cost matrix using the Hungarian algorithm to obtain a first association matching result;

[0014] Among them, the first cost value is a value determined based on the first similarity and the intersection and union ratio between the current moment detection frame and the current moment tracking frame, and the first similarity is the similarity between the target object corresponding to the current moment detection frame and the target object corresponding to the current moment tracking frame.

[0015] Optionally, determining the first perception information of the corresponding first target object based on the updated current-time tracking frame includes:

[0016] Convert the updated two-dimensional coordinates of the bottom center point of the tracking frame at the current moment into world coordinates to obtain target position information of the first target object at the current moment;

[0017] A Kalman filter is used to determine target speed information of the first target object at a current moment based on the target position information, and first perception information of the first target object is determined based on the target position information and the target speed information.

[0018] Optionally, the determining target speed information of the first target object at the current moment based on the target position information by using a Kalman filter includes:

[0019] Inputting the target position information into a Kalman filter to determine the target speed information of the first target object at the current moment based on the initial speed information and the target position information;

[0020] The initial speed information includes an initial lateral speed and an initial longitudinal speed; the initial longitudinal speed is the speed in the direction of the front of the vehicle, and the initial lateral speed is the speed in the direction perpendicular to the front of the vehicle;

[0021] The initial lateral speed is a speed determined based on the lateral position information of the first target object detected by the two-dimensional perception model N frames before; N is a positive integer greater than 1;

[0022] The initial longitudinal speed is a speed determined based on the scale change rate and the longitudinal position information of the first target object detected in the Nth frame of the two-dimensional perception model; the scale change rate is a value calculated based on the height of the tracking frame of the first target object in the previous N frames of the two-dimensional perception model; the lateral position information and the longitudinal position information are both position information in the world coordinate system.

[0023] Optionally, the performing target object detection on the currently acquired image by using a three-dimensional perception model to obtain second perception information of each second target object includes:

[0024] Utilize the three-dimensional perception model to detect the target object in the currently acquired image to obtain the current moment detection amount of each second target object, and associate and match each current moment detection amount with each current moment tracking amount to obtain a second association matching result; the current moment tracking amount is the tracking amount obtained by predicting the tracking amount of the target object at the previous moment based on the three-dimensional perception model; wherein the tracking amount of the target object at the first moment by the three-dimensional perception model is the tracking amount determined based on the detection amount of the target object at the first moment by the three-dimensional perception model;

[0025] Each of the current moment detection quantities and each of the current moment tracking quantities are input into the Kalman filter, so as to utilize the second association matching result and update the corresponding current moment tracking quantity based on the current moment detection quantity, and determine the corresponding second perception information of the second target object based on the updated current moment tracking quantity.

[0026] Optionally, the correlating and matching each of the current moment detection amount and each current moment tracking amount to obtain a second correlating and matching result includes:

[0027] Determine a second cost matrix based on the second generation value between each current moment detection amount and each current moment tracking amount, and use the Hungarian algorithm and based on the second cost matrix to perform association matching on each current moment detection amount and each current moment tracking amount to obtain a second association matching result;

[0028] The second cost value is a value determined based on the distance between the target object corresponding to the detection amount at the current moment and the target object corresponding to the tracking amount at the current moment and the second similarity.

[0029] Optionally, the determining the corresponding second perception information of the second target object based on the updated current tracking amount includes:

[0030] Determine the target speed information of the second target object at the current moment based on the updated current position information in the tracking amount at the current moment;

[0031] If the covariance of the target speed information solved in the Kalman filter is less than a preset value, determining the second perception information of the second target object based on the current position information and the target speed information;

[0032] If the covariance of the target speed information solved in the Kalman filter is not less than a preset value, the speed information at the current moment is determined based on the historical position information of the second target object in a preset historical time period adjacent to the current moment, so as to determine the second perception information of the second target object based on the current position information and the speed information at the current moment;

[0033] The current location information and the historical location information are both location information in a world coordinate system.

[0034] Optionally, determining the speed information at the current moment based on the historical position information of the second target object within a preset historical time period adjacent to the current moment includes:

[0035] Determine the motion vector and rotation matrix of the ego vehicle relative to the origin of the reference coordinate system based on the ego vehicle position information at each moment in the preset historical time period; wherein the reference coordinate system is the ego vehicle coordinate system when the three-dimensional perception model detects the second target object for the first time, and the ego vehicle position information is the position information of the ego vehicle in the world coordinate system;

[0036] Based on the motion vector and the rotation matrix at each moment, the historical position information of the second target object at the corresponding moment is converted into reference position information in the reference coordinate system, and the current moment speed information of the second target object is determined based on each reference position information.

[0037] Optionally, the determining a corresponding second target object to be fused from each of the second targets based on the first target object to be fused includes:

[0038] Determine the angle formed by the first target object to be fused relative to the vehicle based on the detection frame of the first target object to be fused by the two-dimensional perception model to obtain a target angle range;

[0039] Based on the target angle range, the second target object to be fused that meets a second preset condition is determined from each of the second targets; the second preset condition is that the distance between the second target object and the first target object to be fused is less than a preset distance threshold.

[0040] In a second aspect, the present application provides a target object sensing device, comprising:

[0041] An object detection module is used to perform object detection on a current acquired image of the vehicle-mounted monocular camera using a two-dimensional perception model to obtain first perception information of each first object, and to perform object detection on the current acquired image using a three-dimensional perception model to obtain second perception information of each second object;

[0042] a target object determination module, configured to determine, from each of the first targets, a first target object to be fused that satisfies a first preset condition when a first detection distance corresponding to the three-dimensional perception model is less than a second detection distance corresponding to the two-dimensional perception model; the first preset condition being that a distance of the target object relative to the vehicle in a front direction of the vehicle is not greater than the first detection distance;

[0043] A target fusion module is used to determine a corresponding second target to be fused from each of the second targets based on the first target to be fused, and to fuse the perception information corresponding to the first target to be fused and the second target to be fused to obtain final perception information; the second target to be fused and the first target to be fused belong to the same target.

[0044] In a third aspect, the present application provides an electronic device, including:

[0045] Memory, used to store computer programs;

[0046] A processor is used to execute the computer program to implement the aforementioned target object perception method.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which implements the aforementioned target object perception method when executed by a processor.

[0048] In the present application, a two-dimensional perception model is used to perform target detection on the current acquired image of the vehicle-mounted monocular camera to obtain first perception information of each first target object, and a three-dimensional perception model is used to perform target detection on the current acquired image to obtain second perception information of each second target object; when the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, a first target to be fused that meets a first preset condition is determined from each of the first targets; the first preset condition is that the distance of the target relative to the vehicle in the direction of the front of the vehicle is not greater than the first detection distance; based on the first target to be fused, a corresponding second target to be fused is determined from each of the second targets, and the final perception information is obtained by fusing the perception information corresponding to the first target to be fused and the second target to be fused respectively; the second target to be fused and the first target to be fused belong to the same target. It can be seen that the present application detects the target object on the current acquired image of the vehicle-mounted monocular camera through the two-dimensional perception model and the three-dimensional perception model respectively, and obtains the final fused perception information of the target object by fusing the perception information of the target object obtained by the two detections. In this way, the present application can obtain the perception information of the target object from multiple sources by using the vehicle-mounted monocular camera alone. Compared with the combination of multiple sensors to achieve target perception, it not only reduces the cost required for target perception, but also improves the reliability and accuracy of the acquisition of target perception information. In addition, the present application sets the detection distances corresponding to the two-dimensional perception model and the three-dimensional perception model respectively. Since the perception model has a high target detection accuracy within the corresponding detection distance, the target perception information from multiple sources within a smaller detection distance is fused, thereby ensuring the target detection accuracy while improving the perception accuracy of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0050] Figure 1 A flow chart of a target object sensing method disclosed in this application;

[0051] Figure 2 This is an overall flow chart of target object perception disclosed in this application;

[0052] Figure 3 A schematic diagram of a target angle range disclosed in this application;

[0053] Figure 4 A schematic diagram of target object perception disclosed in this application;

[0054] Figure 5 A target object perception flow chart of a two-dimensional perception model disclosed in this application;

[0055] Figure 6 A target object perception flow chart of a specific two-dimensional perception model disclosed in this application;

[0056] Figure 7 A target object perception flow chart of a three-dimensional perception model disclosed in this application;

[0057] Figure 8 A target object perception flow chart of a specific three-dimensional perception model disclosed in this application;

[0058] Fig. 9 This is a schematic diagram of the structure of a target object sensing device disclosed in this application;

[0059] Fig.10 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] For the current two perception schemes of two-dimensional object recognition and three-dimensional object recognition, the perception information of the target object comes from a single perception mode, lacking perception redundancy, that is, when the perception mode is falsely detected, there is no redundant information to avoid these false detections, which ultimately causes abnormal performance of the driving assistance function; and the current perception schemes mostly use a combination of multiple sensors such as cameras and radars to realize object recognition between image layers, and the hardware cost required is relatively high. To this end, the present application provides a method for object perception, which fuses the target object perception information from multiple sources within a small detection distance by using a single on-board monocular camera, thereby ensuring the accuracy of target object detection while improving the accuracy of target object perception and reducing the cost required for target object perception.

[0062] See also Figure 1 As shown, an embodiment of the present invention discloses a target object sensing method, including:

[0063] Step S11: Use a two-dimensional perception model to perform target object detection on the current acquired image of the vehicle-mounted monocular camera to obtain first perception information of each first target object, and use a three-dimensional perception model to perform target object detection on the current acquired image to obtain second perception information of each second target object.

[0064] In this embodiment, for the current acquired image of the vehicle-mounted monocular camera, the current acquired image is respectively input into the two-dimensional perception model and the three-dimensional perception model to perform target detection respectively; wherein, the vehicle-mounted monocular camera is a monocular camera installed on the vehicle, which can be a vehicle-mounted front-view camera or a vehicle-mounted rear-view camera.

[0065] For the two-dimensional perception model, the two-dimensional perception model is used to detect the target object in the current acquired image to obtain the first perception information of each first target object. Specifically, the two-dimensional perception model is used to detect the target object in the current acquired image, and the detection frame of each detected first target object is annotated, so as to obtain the current moment detection frame of each first target object; then each current moment detection frame and each current moment tracking frame are associated and matched to obtain the first association matching result. Among them, the current moment tracking frame is a tracking frame obtained by predicting the tracking frame of the target object at the previous moment based on the two-dimensional perception model, and the tracking frame of the target object at the first moment by the two-dimensional perception model is a tracking frame determined based on the detection frame of the target object at the first moment by the two-dimensional perception model. Further, after obtaining the first association matching result, the corresponding current moment tracking frame is updated based on the current moment detection frame using the first association matching result, so as to determine the first perception information of the corresponding first target object based on the updated current moment tracking frame.

[0066] It should be noted that both the detection frame and the tracking frame have corresponding state vectors; wherein the state vector of the detection frame includes the horizontal and vertical coordinates of the bottom center point of the detection frame, the width and height of the detection frame, and the scale change rate of the detection frame, and the state vector of the detection frame is obtained through detection of a two-dimensional perception model. The state vector of the tracking frame includes the horizontal and vertical coordinates of the bottom center point of the tracking frame, the width and height of the tracking frame, and the scale change rate of the tracking frame, and the state vector of the tracking frame at the current moment is specifically obtained by predicting the state vector of the tracking frame at the previous moment. For example, when predicting the tracking frame at time t-1 to time t, assuming that the tracking frame of the target object moves at a uniform speed in the image, its state transfer equation is: u t =u t-1 +du t-1 ;v t =v t-1 +dv t-1 ;w t =w t-1 ×sc t-1 ;h t =h t-1 ×sc t-1 ;sc t =sc t-1 ; where u represents the horizontal coordinate, v represents the vertical coordinate, w represents the width, h represents the height, sc represents the scale change rate, and t represents the time. It should be noted that when the tracking frame does not fit the scale change rate, the state vector may temporarily ignore the scale change rate. In addition, the state vectors of the detection frame and the tracking frame may also include information such as the target category.

[0067] For the three-dimensional perception model, the target object detection is performed on the current acquired image using the three-dimensional perception model to obtain the second perception information of each second target object. Specifically, the target object detection is performed on the current acquired image using the three-dimensional perception model to obtain the current moment detection amount of each second target object, and the current moment detection amount and the current moment tracking amount are associated and matched to obtain the second association matching result. Wherein, the current moment tracking amount is the tracking amount obtained by predicting the tracking amount of the target object at the previous moment based on the three-dimensional perception model, and the tracking amount of the target object at the first moment by the three-dimensional perception model is the tracking amount determined based on the detection amount of the target object at the first moment by the three-dimensional perception model. Further, after obtaining the second association matching result, each current moment detection amount and each current moment tracking amount are input into the Kalman filter, so as to update the corresponding current moment tracking amount based on the second association matching result and the current moment detection amount, so as to determine the second perception information of the corresponding second target object based on the updated current moment tracking amount. Wherein, both the first perception information and the second perception information may include the position information, velocity information and acceleration information of the target object in the world coordinate system.

[0068] It should be noted that both the detection amount and the tracking amount include the position information of the target object in the world coordinate system (including the horizontal and vertical coordinates), the target object category, the orientation angle (the orientation of the target object relative to the front direction of the vehicle), the length, width and height of the target object, etc.; among them, the detection amount is obtained through detection by a three-dimensional perception model, and the tracking amount at the current moment is obtained by predicting the tracking amount at the previous moment.

[0069] Step S12: When the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, determine the first target object to be fused that meets the first preset condition from each of the first targets; the first preset condition is that the distance of the target object relative to the vehicle in the front direction of the vehicle is not greater than the first detection distance.

[0070] In this embodiment, considering that different perception models correspond to different detection distances, and the detection accuracy of the perception model within the corresponding detection distance is relatively high; therefore, when the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, the first target to be fused that meets the first preset condition can be determined from the first targets detected by the two-dimensional perception model; wherein the first preset condition can be that the distance of the first target relative to the vehicle in the direction of the front of the vehicle is not greater than the first detection distance corresponding to the three-dimensional perception model; the vehicle is a vehicle that performs target object detection and perception information fusion based on the current acquired image of the on-board monocular camera. It should be noted that the detection distance corresponding to the perception model can be set according to the hardware resources. For example, the second detection distance corresponding to the two-dimensional perception model can be set to more than 100 meters, and the first detection distance corresponding to the three-dimensional perception model can be set to within 70 meters.

[0071] Step S13: determining a corresponding second target to be fused from each of the second targets based on the first target to be fused, and fusing the perception information corresponding to the first target to be fused and the second target to be fused to obtain final perception information; the second target to be fused and the first target to be fused belong to the same target.

[0072] In this embodiment, after determining the first target to be fused, a second target to be fused that is the same target as the first target to be fused is determined from the second targets detected by the three-dimensional perception model, and the perception information corresponding to the first target to be fused and the second target to be fused is input into the Kalman filter to fuse the perception information corresponding to the first target to be fused and the second target to be fused, thereby obtaining the final perception information of the target. Figure 2 As shown, the target object perception information obtained by detecting the current acquired image with a two-dimensional perception model and the target object perception information obtained by detecting the current acquired image with a three-dimensional perception model are fused to obtain the final perception information.

[0073] Among them, the perception information includes position information, speed information and acceleration information in the world coordinate system; the position information includes lateral position information and longitudinal position information; the speed information includes longitudinal speed and lateral speed, the longitudinal speed is the speed in the direction of the front of the vehicle, and the lateral speed is the speed in the direction perpendicular to the front of the vehicle.

[0074] According to one of the embodiments, determining the corresponding second target to be fused from each second target based on the first target to be fused may include: determining the angle formed by the first target to be fused relative to the vehicle based on the detection frame of the first target to be fused by the two-dimensional perception model to obtain a target angle range. Then, based on the target angle range, determining the second target to be fused that meets the second preset condition from each second target; wherein the second preset condition is that the distance between the second target and the first target to be fused is less than a preset distance threshold.

[0075] Specifically, Figure 3 As shown, based on the two-dimensional perception model, the detection frame of the first target to be fused determines the left edge point and the right edge point of the detection frame in the vertical direction (y-axis direction) of the vehicle head direction (x-axis direction), wherein the y value of the left edge point is the minimum value of the y values ​​of the frame points in the detection frame, and the y value of the right edge point is the maximum value of the y values ​​of the frame points in the detection frame. Then, based on the connection of the left edge point and the right edge point into a line segment, the target angle range is determined based on the angle formed by the line segment relative to the origin of the vehicle coordinate system; wherein the vehicle coordinate system can be established with the center point of the vehicle as the origin, or with the center point of the rear axle of the vehicle as the origin. After determining the target angle range, it is determined whether there is a second target to be fused in the target angle range, and the distance between the second target detected by the three-dimensional perception model and the first target to be fused is less than the preset distance threshold. If there is, it indicates that the first target to be fused and the second target to be fused belong to the same target, and the perception information corresponding to the first target to be fused and the second target to be fused can be fused to obtain the final perception information.

[0076] It should be noted that if the distance between the second target object to be fused and the first target object to be fused is less than the preset distance threshold, the position information in the perception information corresponding to the second target object to be fused and the position information in the perception information corresponding to the first target object to be fused can be obtained, and the distance between the two position information can be calculated to determine whether it is less than the preset distance threshold. The preset distance threshold can be set according to the actual accuracy requirement.

[0077] In this embodiment, considering that the longitudinal distance measurement of the two-dimensional perception model adopts the pinhole imaging principle, the accuracy of the longitudinal position information may be lower than that of the three-dimensional perception model, which in turn causes the accuracy of the longitudinal speed to be lower than that of the three-dimensional perception model. In addition, since the detection distance of the two-dimensional perception model is greater than that of the three-dimensional perception model, when the three-dimensional perception model detects a certain target object, the two-dimensional perception model may have been tracking it for a period of time, and the lateral speed and lateral position information generally change little, so the accuracy of the lateral position information and lateral speed of the two-dimensional perception model is higher than that of the three-dimensional perception model. Accordingly, after the perception information corresponding to the first target to be fused and the second target to be fused are respectively input into the Kalman filter, the Kalman filter can increase the variance of the longitudinal position information and the longitudinal velocity in the perception information corresponding to the first target to be fused detected by the two-dimensional perception model, so as to set a smaller weight for the longitudinal position information and the longitudinal velocity in the perception information corresponding to the first target to be fused, and increase the variance of the lateral position information and the lateral velocity in the perception information corresponding to the second target to be fused detected by the three-dimensional perception model, so as to set a smaller weight for the lateral position information and the lateral velocity in the perception information corresponding to the second target to be fused, thereby fusing the perception information corresponding to the first target to be fused and the second target to be fused based on the weights, so that the final perception information obtained after the fusion is more dependent on the lateral position information and the lateral velocity of the two-dimensional perception model, and the longitudinal position information and the longitudinal velocity of the three-dimensional perception model, thereby improving the accuracy of the lateral attributes and the longitudinal attributes perceived by the target.

[0078] It should also be noted that when the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, for the first targets detected by the two-dimensional perception model that do not meet the first preset condition, that is, the distance of the target relative to the vehicle in the direction of the front of the vehicle is not greater than the second detection distance and greater than the first detection distance, since these targets temporarily do not have the perception information of the three-dimensional perception model, the perception information of the two-dimensional perception model can be temporarily determined as the final perception information. In the process of continuous movement of the vehicle, if the distance of the target relative to the vehicle in the direction of the front of the vehicle becomes less than the first detection distance, then at this time, the target has both the perception information of the two-dimensional perception model and the perception information of the three-dimensional perception model. In this way, the perception information of the two-dimensional perception model and the perception information of the three-dimensional perception model can be fused to obtain the final fusion information of the target.

[0079] by Figure 4For example, the vehicle coordinate system is established with the center point of the rear axle of the vehicle as the origin, the direction of the front of the vehicle as the x-axis direction, and the vertical direction of the front of the vehicle as the y-axis direction. After the on-board forward-looking monocular camera inputs the current acquired image into the two-dimensional perception model and the three-dimensional perception model respectively, the two-dimensional perception model is used to detect the target object of the current acquired image to obtain the first perception information of each first target object, where Figure 4 The dotted box in the figure is the first target object; at the same time, the three-dimensional perception model is used to detect the target object of the current acquired image to obtain the second perception information of each second target object, where Figure 4 The grayscale filled solid frame in is the second target object; and both the first target object and the second target object include other vehicles, stones, trees, etc. When the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, for the first target object whose distance in the x-axis direction relative to the origin of the vehicle coordinate system is greater than the first detection distance, since the first target object only has the perception information of the two-dimensional perception model, the perception information of the two-dimensional perception model can be temporarily determined as the final perception information of the first target object. For the first target object whose distance in the x-axis direction relative to the origin of the vehicle coordinate system is not greater than the first detection distance, the angle formed by the first target object relative to the origin of the vehicle coordinate system can be determined based on the two-dimensional perception model of the detection frame of the first target object to obtain the target angle range, and then based on the target angle range, a second target object whose distance from the first target object is less than a preset distance threshold is found from each second target object. At this time, it means that the two targets belong to the same target object, that is, the target object has both the perception information of the two-dimensional perception model and the perception information of the three-dimensional perception model. At this time, the perception information of the two-dimensional perception model and the perception information of the three-dimensional perception model can be fused to obtain the final perception information of the target object.

[0080] It can be seen that the present application detects the target object on the current acquired image of the vehicle-mounted monocular camera through the two-dimensional perception model and the three-dimensional perception model respectively, and obtains the final fused perception information of the target object by fusing the perception information of the target object obtained by the two detections. In this way, the present application can obtain the perception information of the target object from multiple sources by using the vehicle-mounted monocular camera alone. Compared with the combination of multiple sensors to achieve target perception, it not only reduces the cost required for target perception, but also improves the reliability and accuracy of the acquisition of target perception information. In addition, the present application sets the detection distances corresponding to the two-dimensional perception model and the three-dimensional perception model respectively. Since the perception model has a high target detection accuracy within the corresponding detection distance, the target perception information from multiple sources within a smaller detection distance is fused, thereby ensuring the target detection accuracy while improving the perception accuracy of the target object.

[0081] Based on the previous embodiment, it can be seen that the previous embodiment describes the complete process of obtaining the final perception information of the same target object by fusing the perception information of the two-dimensional perception model and the perception information of the three-dimensional perception model. Next, the embodiment of the present invention will elaborate on how to obtain the perception information of the target object through the two-dimensional perception model. Figure 5 As shown, the embodiment of the present invention discloses a target object perception process of a two-dimensional perception model, including:

[0082] Step S21, use the two-dimensional perception model to detect the target object in the currently acquired image to obtain the current moment detection frame of each first target object, and associate and match each current moment detection frame with each current moment tracking frame to obtain a first association matching result; the current moment tracking frame is a tracking frame obtained by predicting the tracking frame of the target object at the previous moment based on the two-dimensional perception model; wherein, the tracking frame of the target object at the first moment by the two-dimensional perception model is a tracking frame determined based on the detection frame of the target object at the first moment by the two-dimensional perception model.

[0083] In this embodiment, Figure 6 As shown, the currently acquired image is input into the two-dimensional perception model so as to detect the target object on the currently acquired image using the two-dimensional perception model, thereby obtaining the current moment detection frame of each first target object, and performing association matching on each current moment detection frame and each current moment tracking frame to obtain a corresponding first association matching result. Wherein, both the detection frame and the tracking frame have corresponding state vectors.

[0084] According to one of the embodiments, each current moment detection frame and each current moment tracking frame are associated and matched to obtain a first associated matching result, which may specifically include: calculating the first generation value between each current moment detection frame and each current moment tracking frame, and determining the first cost matrix based on each first generation value, and then using the Hungarian algorithm and based on the first cost matrix to associate and match each current moment detection frame and each current moment tracking frame to obtain the first associated matching result. Among them, the first generation value is mainly based on two parts, one is the first similarity between the target object corresponding to the current moment detection frame and the target object corresponding to the current moment tracking frame, and the other is the intersection over union (IoU) between the current moment detection frame and the current moment tracking frame.

[0085] For the first similarity between the target object corresponding to the detection frame at the current moment and the target object corresponding to the tracking frame at the current moment, the classic cosine similarity can be used, because the value of cosine similarity is [0,1], which is in the same range as the intersection over union (IoU), that is, the values ​​of the first similarity and the intersection over union (IoU) are both between 0 and 1. Specifically, the first similarity is calculated based on the feature vector of the target object corresponding to the detection frame at the current moment and the feature vector of the target object corresponding to the tracking frame at the current moment; wherein the feature vector is given by the last layer feature map of the two-dimensional perception model, and the length of the feature vector is in direct proportion to the hardware computing resources. When the hardware computing resources are sufficient, the length of the feature vector is longer, and accordingly, the description of the target object is more accurate. It should be pointed out that the specific calculation formula for the first similarity is: similarity1=featureVec1×featureVec2 / (norm(featureVec1)×norm(featureVec2)); where similarity1 represents the first similarity, featureVec1 represents the target feature vector corresponding to the detection frame at the current moment, featureVec2 represents the target feature vector corresponding to the tracking frame at the current moment, and norm() is a function used to calculate the length of the feature vector.

[0086] Correspondingly, for the first generation value, a first weight parameter is set for the first similarity, and a second weight parameter is set for the intersection-over-union ratio, wherein the first weight parameter and the second weight parameter are added to one, thereby weighted calculation of the first similarity and the intersection-over-union ratio is performed based on the weight parameters to obtain the first generation value. It should be pointed out that the specific calculation formula of the first generation value is: cost1=a×IoU+(1-a)×similarity1; wherein a represents the weight parameter, IoU represents the intersection-over-union ratio, similarity1 represents the first similarity, and cost1 represents the first generation value. The specific value of the weight parameter a can be determined according to the accuracy of the target object feature vector.

[0087] Step S22: using the first association matching result and based on the current moment detection frame, updating the corresponding current moment tracking frame, and converting the two-dimensional coordinates of the bottom center point of the updated current moment tracking frame into world coordinates to obtain the target position information of the first target object at the current moment.

[0088] In this embodiment, after determining the first association matching result, the first association matching result, the state vector of the current moment detection frame and the state vector of the current moment tracking frame are input into the Kalman filter, so that the Kalman filter uses the first association matching result and based on the current moment detection frame to update the corresponding current moment tracking frame to obtain an updated current moment tracking frame.

[0089] Specifically, Figure 6 As shown, the Kalman filter utilizes the first association matching result and updates the corresponding current moment tracking frame based on the current moment detection frame. Specifically, it may include: when the current moment detection frame and the current moment tracking frame match, the Kalman filter updates the state vector of the current moment tracking frame based on the state vector of the current moment detection frame, thereby optimizing the state vector of the current moment tracking frame; when there is no matching current moment tracking frame for the current moment detection frame, a new current moment tracking frame is created based on the current moment detection frame, and the state vector of the current moment detection frame is set to the state vector of the current moment tracking frame; when there is no matching current moment detection frame for the current moment tracking frame, the current moment tracking frame is maintained, that is, the state vector of the current moment tracking frame is retained.

[0090] Further, such as Figure 6 As shown, after the Kalman filter completes the update of the tracking frame at the current moment, the two-dimensional coordinates (i.e., the horizontal coordinate and the vertical coordinate) of the bottom center point of the updated tracking frame at the current moment are converted into world coordinates based on the classic monocular camera pinhole imaging principle to obtain the target position information of the first target object at the current moment; that is, the target position information is the position information in the world coordinate system.

[0091] Step S23: using a Kalman filter and based on the target position information, determining the target speed information of the first target object at the current moment, and determining the first perception information of the first target object based on the target position information and the target speed information.

[0092] In this embodiment, Figure 6 As shown, when using the Kalman filter to determine the target speed information of the first target object at the current moment based on the target position information, the target speed information of the first target object at the current moment can be determined based on the target position information by using the association between the position information and the speed information, or the target speed information of the first target object at the current moment can be determined based on the target position information by using the initial speed information. After that, the first perception information of the first target object can be determined based on the target position information and the target speed information of the first target object at the current moment. In addition, when the perception information includes acceleration information, the acceleration information of the target object at the current moment can also be determined based on the target position information by using the association between the position information and the acceleration information through the Kalman filter.

[0093] It should be noted that the initial speed information includes an initial lateral speed and an initial longitudinal speed; wherein the initial longitudinal speed is the speed in the direction of the front of the vehicle, and the initial lateral speed is the speed in the direction perpendicular to the front of the vehicle.

[0094] The initial lateral speed is specifically determined based on the lateral position information of the first target object detected in the first N frames of the two-dimensional perception model; wherein N is a positive integer greater than 1. The initial longitudinal speed is specifically determined based on the pre-calculated scale change rate and the longitudinal position information of the first target object detected in the Nth frame of the two-dimensional perception model; wherein the scale change rate is a value calculated based on the height of the tracking frame of the first target object in the first N frames of the two-dimensional perception model. It should be noted that both the lateral position information and the longitudinal position information are position information in the world coordinate system.

[0095] The calculation formula of initial lateral velocity is: The calculation formula of the initial longitudinal velocity is: vx=(1-sc)×dx / △t. Where vy represents the initial lateral velocity and N represents the total number of frames. Indicates The horizontal position information of the first target object at the time of the frame, Indicates The time information of the frame. vx represents the initial longitudinal velocity, sc represents the pre-calculated scale change rate, dx represents the longitudinal position information of the first target object detected by the two-dimensional perception model in the Nth frame, and △t represents the change per unit time, that is, the time interval between two frames.

[0096] For the pre-calculated scale change rate, after creating a new current-time tracking frame based on the current-time detection frame of the first target object, the height of the tracking frame of the first target object in the first N frames of the two-dimensional perception model can be cached, and the initial scale change rate can be calculated based on the height of the tracking frame of the first target object in the first N frames; wherein the value of N can be adjusted according to the requirements of the response speed and accuracy of the target object perception. Taking N=5 as an example, the initial scale change rate sc=sum(h2 / h1,h3 / h2,h4 / h3,h5 / h4) / 4, sum() is a function for calculating the sum of values, and h represents the height of the tracking frame.

[0097] It should be noted that after calculating the initial scale change rate of the tracking frame, the Kalman filter needs to update the state vector of the current tracking frame based on the state vector of the current detection frame when the current detection frame matches the current tracking frame, where the state vector includes the scale change rate. Therefore, the scale change rate in the state vector of the current detection frame is a value determined based on the height of the current detection frame and the height of the previous detection frame.

[0098] It can be seen that the embodiment of the present invention detects the target object in the current image acquired by the vehicle-mounted monocular camera through the two-dimensional perception model, and performs association matching on the current moment detection frame of the target object and the current moment tracking frame obtained by predicting the tracking frame at the previous moment, so as to utilize the Kalman filter and update the current moment tracking frame based on the association matching result to improve the accuracy of target tracking; further, the embodiment of the present invention can obtain target object perception information from multiple sources through the two-dimensional perception model and the three-dimensional perception model based on the vehicle-mounted monocular camera, and by fusing the target object perception information from multiple sources, it can reduce the cost required for target object perception while improving the reliability and accuracy of target object perception information acquisition.

[0099] Based on the previous embodiment, it can be seen that the previous embodiment describes how to obtain the perception information of the target object through a two-dimensional perception model. Next, the embodiment of the present invention will elaborate on how to obtain the perception information of the target object through a three-dimensional perception model. Figure 7 As shown, the embodiment of the present invention discloses a target object perception process of a three-dimensional perception model, including:

[0100] Step S31, use the three-dimensional perception model to detect the target object in the currently acquired image to obtain the current moment detection amount of each second target object, and associate and match each current moment detection amount with each current moment tracking amount to obtain a second association matching result; the current moment tracking amount is the tracking amount obtained by predicting the tracking amount of the target object at the previous moment based on the three-dimensional perception model; wherein the tracking amount of the target object at the first moment by the three-dimensional perception model is the tracking amount determined based on the detection amount of the target object at the first moment by the three-dimensional perception model.

[0101] In this embodiment, Figure 8 As shown, the currently acquired image is input into the three-dimensional perception model so that the three-dimensional perception model can be used to perform target object detection on the currently acquired image to obtain the current moment detection amount of each second target object, and each current moment detection amount and each current moment tracking amount are associated and matched to obtain a corresponding second associated matching result.

[0102] According to one of the embodiments, each current moment detection quantity and each current moment tracking quantity are associated and matched to obtain a second associated matching result, including: calculating the second generation value between each current moment detection quantity and each current moment tracking quantity, and determining the second cost matrix based on each second generation value, and then using the Hungarian algorithm and based on the second cost matrix to associate and match each current moment detection quantity and each current moment tracking quantity, so as to obtain the second associated matching result. Among them, the second generation value mainly includes two parts, one part is the distance between the target object corresponding to the current moment detection quantity and the target object corresponding to the current moment tracking quantity, and the other part is the second similarity between the target object corresponding to the current moment detection quantity and the target object corresponding to the current moment tracking quantity.

[0103] Among them, for the distance between the target object corresponding to the current moment detection amount and the target object corresponding to the current moment tracking amount, the Euclidean distance can be used. For the second similarity between the target object corresponding to the current moment detection amount and the target object corresponding to the current moment tracking amount, the cosine similarity can be used, that is, the second similarity is calculated based on the target object feature vector corresponding to the current moment detection amount and the target object feature vector corresponding to the current moment tracking amount, wherein the feature vector is composed of the length, width, height and orientation angle of the target object.

[0104] As for the second generation value, the first weight parameter can be set for the distance, and the second weight parameter can be set for the second similarity, wherein the first weight parameter and the second weight parameter are added to one, thereby weighted calculation of the distance and the second similarity is performed based on the weight parameters to obtain the second generation value. It should be pointed out that the specific calculation formula of the second generation value is: cost2=b×(1 / distance)+(1-b)×similarity2; wherein b represents the weight parameter, distance represents the distance, similarity2 represents the second similarity, and cost2 represents the second generation value.

[0105] Step S32, input each of the current moment detection quantities and each of the current moment tracking quantities into the Kalman filter, so as to utilize the second association matching result and update the corresponding current moment tracking quantity based on the current moment detection quantity, and determine the corresponding second perception information of the second target object based on the updated current moment tracking quantity.

[0106] In this embodiment, after obtaining the second association matching result, each current moment detection amount and each current moment tracking amount are input into the Kalman filter, so that the Kalman filter updates the corresponding current moment tracking amount based on the second association matching result and the current moment detection amount, and solves the speed information of the second target at the current moment based on the current position information in the updated current moment tracking amount, so as to obtain the second perception information of the second target based on the current position information and the speed information at the current moment. Wherein, the current position information is the position information in the world coordinate system, including the horizontal coordinate and the vertical coordinate.

[0107] Specifically, Figure 8 As shown, the Kalman filter uses the second association matching result and updates the corresponding current moment tracking amount based on the current moment detection amount, which can specifically include: when the current moment detection amount and the current moment tracking amount match, the Kalman filter updates the current moment tracking amount based on the current moment detection amount, thereby optimizing the current moment tracking amount; when there is no matching current moment tracking amount for the current moment detection amount, a new current moment tracking amount is created based on the current moment detection amount; when there is no matching current moment detection amount for the current moment tracking amount, the current moment tracking amount is maintained, that is, the current moment tracking amount is retained. After completing the update of the current moment tracking amount, the second perception information of the corresponding second target object is determined based on the updated current moment tracking amount through the Kalman filter.

[0108] When the Kalman filter solves the speed information of the second target object at the current moment based on the current position information, the target speed information of the second target object at the current moment can be solved by using the Kalman filter based on the current position information; if the covariance of the target speed information solved in the Kalman filter is less than the preset value, it indicates that the target speed information solved by the Kalman filter is relatively stable and accurate, so the second perception information of the second target object can be directly determined based on the current position information and the target speed information. If the covariance of the target speed information solved in the Kalman filter is not less than the preset value, it indicates that the target speed information solved by the Kalman filter is not very stable, so the current moment speed information can be determined based on the historical position information of the second target object in the preset historical time period adjacent to the current moment, and the historical position information is the position information in the world coordinate system, including the horizontal coordinate and the vertical coordinate, so as to determine the second perception information of the second target object based on the current position information and the current moment speed information. Among them, the preset historical time period may include the current moment, or may not include the current moment.

[0109] Among them, for determining the speed information at the current moment based on the historical position information of the second target object within a preset historical time period adjacent to the current moment, it includes: determining the motion vector and rotation matrix of the vehicle relative to the origin of the reference coordinate system based on the vehicle position information at each moment in the preset historical time period; wherein the reference coordinate system is the vehicle coordinate system when the three-dimensional perception model detects the second target object for the first time, and the vehicle position information is the position information of the vehicle in the world coordinate system. Then, based on the motion vector and rotation matrix at each moment, the historical position information of the second target object at the corresponding moment is converted into the reference position information in the reference coordinate system, and the speed information of the second target object at the current moment is determined based on each reference position information. The specific formulas involved are as follows:

[0110] ;

[0111] ;

[0112] Among them, s represents the motion vector, R represents the rotation matrix, Represents the historical position information of the second target in the world coordinate system, represents the reference position information of the second target in the reference coordinate system, Indicates a moment, Indicates the total number of moments in the preset historical time period. Indicates the current speed information including the longitudinal speed and the lateral speed, wherein the longitudinal speed is the speed in the direction of the front of the vehicle, and the lateral speed is the speed in the direction perpendicular to the front of the vehicle.

[0113] It should also be noted that, for the case where the covariance when solving the target speed information in the Kalman filter is not less than a preset value, after determining the current moment speed information based on the historical position information of the second target object within a preset historical time period adjacent to the current moment, the Kalman filter can also be maintained based on the current moment speed information, so that the Kalman filter can use the current moment speed information as the initial speed information to achieve subsequent solution of the target object speed information, thereby achieving correction and maintenance of the Kalman filter.

[0114] It can be seen that the embodiment of the present invention detects the target object in the current image acquired by the vehicle-mounted monocular camera through the three-dimensional perception model, and associates and matches the current moment detection amount of the target object and the current moment tracking amount obtained by predicting the tracking amount at the previous moment, thereby utilizing the Kalman filter and updating the current moment tracking amount based on the association matching result to improve the accuracy of target tracking; further, the embodiment of the present invention can obtain target object perception information from multiple sources through two-dimensional perception models and three-dimensional perception models based on the vehicle-mounted monocular camera, and by fusing target object perception information from multiple sources, it can reduce the cost required for target object perception while improving the reliability and accuracy of target object perception information acquisition.

[0115] See also Fig. 9 As shown, the embodiment of the present invention discloses a target object sensing device, including

[0116] The object detection module 11 is used to perform object detection on the current acquired image of the vehicle-mounted monocular camera using a two-dimensional perception model to obtain first perception information of each first object, and to perform object detection on the current acquired image using a three-dimensional perception model to obtain second perception information of each second object;

[0117] The target object determination module 12 is used to determine a first target object to be fused that satisfies a first preset condition from each of the first targets when the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model; the first preset condition is that the distance of the target object relative to the vehicle in the direction of the front of the vehicle is not greater than the first detection distance;

[0118] The target object fusion module 13 is used to determine the corresponding second target object to be fused from each of the second targets based on the first target object to be fused, and to fuse the perception information corresponding to the first target object to be fused and the second target object to be fused to obtain the final perception information; the second target object to be fused and the first target object to be fused belong to the same target object.

[0119] It can be seen that the present application detects the target object on the current acquired image of the vehicle-mounted monocular camera through the two-dimensional perception model and the three-dimensional perception model respectively, and obtains the final fused perception information of the target object by fusing the perception information of the target object obtained by the two detections. In this way, the present application can obtain the perception information of the target object from multiple sources by using the vehicle-mounted monocular camera alone. Compared with the combination of multiple sensors to achieve target perception, it not only reduces the cost required for target perception, but also improves the reliability and accuracy of the acquisition of target perception information. In addition, the present application sets the detection distances corresponding to the two-dimensional perception model and the three-dimensional perception model respectively. Since the perception model has a high target detection accuracy within the corresponding detection distance, the target perception information from multiple sources within a smaller detection distance is fused, thereby ensuring the target detection accuracy while improving the perception accuracy of the target object.

[0120] In some specific embodiments, the target detection module 11 includes:

[0121] A first association matching submodule is used to perform target object detection on the currently acquired image using a two-dimensional perception model to obtain a current moment detection frame for each first target object, and to perform association matching on each current moment detection frame and each current moment tracking frame to obtain a first association matching result; the current moment tracking frame is a tracking frame obtained by predicting the tracking frame of the target object at a previous moment based on the two-dimensional perception model; wherein the tracking frame of the target object at the first moment by the two-dimensional perception model is a tracking frame determined based on the detection frame of the target object at the first moment by the two-dimensional perception model;

[0122] The tracking frame updating submodule is used to update the corresponding current moment tracking frame based on the current moment detection frame using the first association matching result, and determine the first perception information of the corresponding first target object based on the updated current moment tracking frame.

[0123] In some specific embodiments, the first association matching submodule includes:

[0124] A first cost matrix determining unit is used to determine a first cost matrix based on first generation values ​​between each current moment detection frame and each current moment tracking frame, and to perform association matching on each current moment detection frame and each current moment tracking frame based on the first cost matrix using a Hungarian algorithm to obtain a first association matching result;

[0125] Among them, the first cost value is a value determined based on the first similarity and the intersection and union ratio between the current moment detection frame and the current moment tracking frame, and the first similarity is the similarity between the target object corresponding to the current moment detection frame and the target object corresponding to the current moment tracking frame.

[0126] In some specific embodiments, the tracking frame updating submodule includes:

[0127] A coordinate conversion unit, used for converting the updated two-dimensional coordinates of the bottom center point of the tracking frame at the current moment into world coordinates to obtain target position information of the first target object at the current moment;

[0128] The first perception information determination unit is used to determine the target speed information of the first target object at a current moment based on the target position information using a Kalman filter, and to determine the first perception information of the first target object based on the target position information and the target speed information.

[0129] In some specific embodiments, the first perception information determining unit includes:

[0130] a speed information determining unit, configured to input the target position information into a Kalman filter, so as to determine the target speed information of the first target object at a current moment based on the initial speed information and the target position information;

[0131] The initial speed information includes an initial lateral speed and an initial longitudinal speed; the initial longitudinal speed is the speed in the direction of the front of the vehicle, and the initial lateral speed is the speed in the direction perpendicular to the front of the vehicle;

[0132] The initial lateral speed is a speed determined based on the lateral position information of the first target object detected by the two-dimensional perception model N frames before; N is a positive integer greater than 1;

[0133] The initial longitudinal speed is a speed determined based on the scale change rate and the longitudinal position information of the first target object detected in the Nth frame of the two-dimensional perception model; the scale change rate is a value calculated based on the height of the tracking frame of the first target object in the previous N frames of the two-dimensional perception model; the lateral position information and the longitudinal position information are both position information in the world coordinate system.

[0134] In some specific embodiments, the target detection module 11 includes:

[0135] A second association matching submodule is used to perform target object detection on the currently acquired image using a three-dimensional perception model to obtain a current moment detection amount of each second target object, and to perform association matching on each current moment detection amount and each current moment tracking amount to obtain a second association matching result; the current moment tracking amount is a tracking amount obtained by predicting the tracking amount of the target object at a previous moment based on the three-dimensional perception model; wherein the tracking amount of the target object at a first moment by the three-dimensional perception model is a tracking amount determined based on the detection amount of the target object at the first moment by the three-dimensional perception model;

[0136] A tracking quantity update submodule is used to input each current moment detection quantity and each current moment tracking quantity into a Kalman filter, so as to update the corresponding current moment tracking quantity based on the current moment detection quantity using the second association matching result, and determine the corresponding second perception information of the second target object based on the updated current moment tracking quantity.

[0137] In some specific embodiments, the second association matching submodule includes:

[0138] A second cost matrix determining unit is used to determine a second cost matrix based on the second generation value between each current moment detection amount and each current moment tracking amount, and to perform association matching on each current moment detection amount and each current moment tracking amount based on the second cost matrix using the Hungarian algorithm to obtain a second association matching result;

[0139] The second cost value is a value determined based on the distance between the target object corresponding to the detection amount at the current moment and the target object corresponding to the tracking amount at the current moment and the second similarity.

[0140] In some specific embodiments, the tracking amount update submodule includes:

[0141] A speed information solving unit, used for solving the target speed information of the second target object at the current moment based on the updated current position information in the tracking amount at the current moment;

[0142] A second perception information determination unit is used to determine the second perception information of the second target object based on the current position information and the target speed information if the covariance when the target speed information is solved in the Kalman filter is less than a preset value; if the covariance when the target speed information is solved in the Kalman filter is not less than a preset value, determine the current moment speed information based on the historical position information of the second target object in a preset historical time period adjacent to the current moment, so as to determine the second perception information of the second target object based on the current position information and the current moment speed information; wherein the current position information and the historical position information are both position information in the world coordinate system.

[0143] In some specific embodiments, the second perception information determining unit includes:

[0144] A coordinate system parameter determination unit, configured to determine a motion vector and a rotation matrix of the ego vehicle relative to an origin of a reference coordinate system based on the ego vehicle position information at each moment in the preset historical time period; wherein the reference coordinate system is the ego vehicle coordinate system when the three-dimensional perception model detects the second target object for the first time, and the ego vehicle position information is the position information of the ego vehicle in the world coordinate system;

[0145] A position information conversion unit is used to convert the historical position information of the second target object at a corresponding moment into reference position information in the reference coordinate system based on the motion vector and the rotation matrix at each moment, and determine the current moment speed information of the second target object based on each reference position information.

[0146] In some specific embodiments, the target fusion module 13 includes:

[0147] an angle range acquisition unit, configured to determine an angle formed by the first target object to be fused relative to the vehicle based on the detection frame of the first target object to be fused by the two-dimensional perception model, so as to obtain a target angle range;

[0148] A target object to be fused determining unit is used to determine the second target object to be fused that meets a second preset condition from each second target object based on the target angle range; the second preset condition is that the distance between the second target object and the first target object to be fused is less than a preset distance threshold.

[0149] Furthermore, the present application also discloses an electronic device. Fig.10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0150] Fig.10A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the target object perception method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0151] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0152] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0153] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the target object sensing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0154] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the target object sensing method disclosed above. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0155] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0156] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0157] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0158] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0159] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A target object perception method, characterized in that: include: Performing target object detection on a current acquired image of the vehicle-mounted monocular camera using a two-dimensional perception model to obtain first perception information of each first target object, and performing target object detection on the current acquired image using a three-dimensional perception model to obtain second perception information of each second target object; When the first detection distance corresponding to the three-dimensional perception model is less than the second detection distance corresponding to the two-dimensional perception model, determining a first target object to be fused that meets a first preset condition from each of the first targets; the first preset condition is that the distance of the target object relative to the vehicle in the direction of the front of the vehicle is not greater than the first detection distance; Determining a corresponding second target to be fused from each of the second targets based on the first target to be fused, and fusing the perception information corresponding to the first target to be fused and the second target to be fused to obtain final perception information; The second target object to be fused and the first target object to be fused are the same target object; Wherein, determining the corresponding second target object to be fused from each of the second targets based on the first target object to be fused includes: Determine the angle formed by the first target object to be fused relative to the vehicle based on the detection frame of the first target object to be fused by the two-dimensional perception model to obtain a target angle range; Based on the target angle range, the second target object to be fused that meets a second preset condition is determined from each of the second targets; the second preset condition is that the distance between the second target object and the first target object to be fused is less than a preset distance threshold.

2. The target object sensing method according to claim 1, characterized in that: The method of using the two-dimensional perception model to perform target object detection on the current acquired image of the vehicle-mounted monocular camera to obtain first perception information of each first target object includes: Detecting an object in the currently acquired image using a two-dimensional perception model to obtain a current moment detection frame for each first object, and performing association matching on each current moment detection frame and each current moment tracking frame to obtain a first association matching result; the current moment tracking frame is a tracking frame obtained by predicting a tracking frame of the object at a previous moment based on the two-dimensional perception model; wherein the tracking frame of the object at the first moment by the two-dimensional perception model is a tracking frame determined based on the detection frame of the object at the first moment by the two-dimensional perception model; The corresponding current moment tracking frame is updated based on the current moment detection frame using the first association matching result, and the first perception information of the corresponding first target object is determined based on the updated current moment tracking frame.

3. The target object sensing method according to claim 2, characterized in that: The associating and matching each current moment detection frame and each current moment tracking frame to obtain a first associating and matching result includes: Determine a first cost matrix based on the first generation value between each current moment detection frame and each current moment tracking frame, and perform association matching on each current moment detection frame and each current moment tracking frame based on the first cost matrix using the Hungarian algorithm to obtain a first association matching result; Among them, the first cost value is a value determined based on the first similarity and the intersection and union ratio between the current moment detection frame and the current moment tracking frame, and the first similarity is the similarity between the target object corresponding to the current moment detection frame and the target object corresponding to the current moment tracking frame.

4. The target object sensing method according to claim 2, characterized in that: The determining the first perception information of the corresponding first target object based on the updated current moment tracking frame includes: Convert the updated two-dimensional coordinates of the bottom center point of the tracking frame at the current moment into world coordinates to obtain target position information of the first target object at the current moment; A Kalman filter is used to determine target speed information of the first target object at a current moment based on the target position information, and first perception information of the first target object is determined based on the target position information and the target speed information.

5. The target object sensing method according to claim 4, characterized in that: The method of using a Kalman filter and determining target speed information of the first target object at a current moment based on the target position information includes: Inputting the target position information into a Kalman filter to determine the target speed information of the first target object at the current moment based on the initial speed information and the target position information; The initial speed information includes an initial lateral speed and an initial longitudinal speed; the initial longitudinal speed is the speed in the direction of the front of the vehicle, and the initial lateral speed is the speed in the direction perpendicular to the front of the vehicle; The initial lateral speed is a speed determined based on the lateral position information of the first target object detected by the two-dimensional perception model N frames before; N is a positive integer greater than 1; The initial longitudinal speed is a speed determined based on the scale change rate and the longitudinal position information of the first target object detected in the Nth frame of the two-dimensional perception model; the scale change rate is a value calculated based on the height of the tracking frame of the first target object in the previous N frames of the two-dimensional perception model; the lateral position information and the longitudinal position information are both position information in the world coordinate system.

6. The target object sensing method according to claim 1, characterized in that: The using the three-dimensional perception model to perform target object detection on the currently acquired image to obtain second perception information of each second target object includes: Utilize the three-dimensional perception model to detect the target object in the currently acquired image to obtain the current moment detection amount of each second target object, and associate and match each current moment detection amount with each current moment tracking amount to obtain a second association matching result; the current moment tracking amount is the tracking amount obtained by predicting the tracking amount of the target object at the previous moment based on the three-dimensional perception model; wherein the tracking amount of the target object at the first moment by the three-dimensional perception model is the tracking amount determined based on the detection amount of the target object at the first moment by the three-dimensional perception model; Each of the current moment detection quantities and each of the current moment tracking quantities are input into the Kalman filter, so as to utilize the second association matching result and update the corresponding current moment tracking quantity based on the current moment detection quantity, and determine the corresponding second perception information of the second target object based on the updated current moment tracking quantity.

7. The target object sensing method according to claim 6, characterized in that: The correlating and matching each of the current moment detection amount and each current moment tracking amount to obtain a second correlating and matching result includes: Determine a second cost matrix based on the second generation value between each current moment detection amount and each current moment tracking amount, and use the Hungarian algorithm and based on the second cost matrix to perform association matching on each current moment detection amount and each current moment tracking amount to obtain a second association matching result; The second cost value is a value determined based on the distance between the target object corresponding to the detection amount at the current moment and the target object corresponding to the tracking amount at the current moment and the second similarity.

8. The target object sensing method according to claim 6, characterized in that: The determining the corresponding second perception information of the second target object based on the updated current tracking amount includes: Determine the target speed information of the second target object at the current moment based on the updated current position information in the tracking amount at the current moment; If the covariance of the target speed information solved in the Kalman filter is less than a preset value, determining the second perception information of the second target object based on the current position information and the target speed information; If the covariance of the target speed information solved in the Kalman filter is not less than a preset value, the speed information at the current moment is determined based on the historical position information of the second target object in a preset historical time period adjacent to the current moment, so as to determine the second perception information of the second target object based on the current position information and the speed information at the current moment; The current location information and the historical location information are both location information in a world coordinate system.

9. The target object sensing method according to claim 8, characterized in that: The determining the speed information at the current moment based on the historical position information of the second target object within a preset historical time period adjacent to the current moment includes: Determine the motion vector and rotation matrix of the ego vehicle relative to the origin of the reference coordinate system based on the ego vehicle position information at each moment in the preset historical time period; wherein the reference coordinate system is the ego vehicle coordinate system when the three-dimensional perception model detects the second target object for the first time, and the ego vehicle position information is the position information of the ego vehicle in the world coordinate system; Based on the motion vector and the rotation matrix at each moment, the historical position information of the second target object at the corresponding moment is converted into reference position information in the reference coordinate system, and the current moment speed information of the second target object is determined based on each reference position information.

10. A target object sensing device, characterized in that: include: An object detection module is used to perform object detection on a current acquired image of the vehicle-mounted monocular camera using a two-dimensional perception model to obtain first perception information of each first object, and to perform object detection on the current acquired image using a three-dimensional perception model to obtain second perception information of each second object; a target object determination module, configured to determine, from each of the first targets, a first target object to be fused that satisfies a first preset condition when a first detection distance corresponding to the three-dimensional perception model is less than a second detection distance corresponding to the two-dimensional perception model; the first preset condition being that a distance of the target object relative to the vehicle in a front direction of the vehicle is not greater than the first detection distance; a target object fusion module, configured to determine a corresponding second target object to be fused from each of the second targets based on the first target object to be fused, and to fuse the perception information corresponding to the first target object to be fused and the second target object to be fused to obtain final perception information; The second target object to be fused and the first target object to be fused are the same target object; Among them, the target object fusion module is specifically used to: determine the angle formed by the first target object to be fused relative to the vehicle based on the detection frame of the first target object to be fused by the two-dimensional perception model to obtain the target angle range; based on the target angle range, determine the second target object to be fused that meets the second preset condition from each second target object; the second preset condition is that the distance between the second target object and the first target object to be fused is less than a preset distance threshold.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the target object perception method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the target object perception method as described in any one of claims 1 to 9.

Images