Obstacle detection method, controller and vehicle

Abstract

This application provides an obstacle detection method, a controller, and a vehicle. The method includes: the controller inputting the posterior information of a first obstacle at a first moment into a prior motion model to estimate the prior state information of the first obstacle at a second moment. The controller can use a preset obstacle recognition algorithm to detect second obstacles in a second image at the second moment and acquire target detection information for each second obstacle. The controller can determine the matching relationship between the first obstacle and the second obstacle based on the matching degree between the prior state information and the target detection information. This matching degree can be determined by calculating the two-dimensional overlap and the three-dimensional residual value. The controller can use a preset filtering algorithm to fuse the target detection information of the second obstacle with the prior state information of the first obstacle that matches the second obstacle to obtain the posterior information of the second obstacle. The method of this application improves the estimation accuracy of obstacle information.

Description

Obstacle detection method, controller and vehicle Technical Field

[0001] The present application relates to communication technology, and in particular to an obstacle detection method, a controller, and a vehicle. Background Art

[0002] With the development of science and technology, intelligent driving technology is increasingly being applied to vehicles. In the intelligent driving process, detecting obstacles such as vehicles, pedestrians, and objects around the vehicle is a very important step.

[0003] In the prior art, a vehicle may be provided with at least one camera around it, and a controller disposed inside the vehicle may detect obstacles around the vehicle by identifying image information uploaded by the camera.

[0004] However, obstacle detection based on the above method may have a large detection error problem for obstacles at medium and long distances.

[0005] Summary of the Invention

[0006] The present application provides an obstacle detection method, a controller, and a vehicle to solve the problem of large obstacle detection errors.

[0007] In a first aspect, the present application provides an obstacle detection method, comprising:

[0008] Inputting a priori information of each first obstacle in the first image at a first moment into a priori motion model to estimate priori state information of the first obstacle at a second moment, the priori state information including an estimated position of the first obstacle at the second moment, and the a priori information including a predicted position of the first obstacle at the first moment, where the first moment and the second moment are adjacent previous moments;

[0009] identifying, based on a second image captured by a camera on the vehicle at a second moment, target detection information of a second obstacle in the second image at the second moment, the target detection information including a detected position of the second obstacle;

[0010] determining, based on the priori state information of the first obstacle and the target detection information of the second obstacle, a first obstacle that matches each second obstacle;

[0011] Based on a preset filtering algorithm, the target detection information of the second obstacle and the priori state information of the first obstacle that matches the second obstacle are fused to obtain the posterior information of the second obstacle.

[0012] Optionally, the identifying and obtaining target detection information of a second obstacle in the second image at the second moment based on the second image captured by a camera on the vehicle at the second moment specifically includes:

[0013] identifying the second obstacles in the second image using a two-dimensional recognition algorithm, and obtaining a two-dimensional detection position of each second obstacle, wherein the two-dimensional detection position includes a two-dimensional positioning coordinate, a two-dimensional width, and a two-dimensional length of a two-dimensional detection frame of the second obstacle;

[0014] determining a first distance in the two-dimensional detection position according to the two-dimensional positioning coordinates, the two-dimensional width, the two-dimensional length, and camera information of the camera on the vehicle in the two-dimensional detection position;

[0015] identifying the second obstacles in the second image using a three-dimensional recognition algorithm to obtain a three-dimensional detected position of each second obstacle, the three-dimensional detected position including a center point coordinate, a three-dimensional width, a three-dimensional length, and a three-dimensional height of the second obstacle in vehicle coordinates;

[0016] A second distance in the three-dimensional detection position is determined based on the center point coordinates, the three-dimensional width, the three-dimensional length, and the three-dimensional height in the three-dimensional detection position.

[0017] Optionally, the determining, according to the priori state information of the first obstacle and the target detection information of the second obstacle, a first obstacle matching each second obstacle specifically includes:

[0018] Calculating a degree of overlap between each first obstacle and each second obstacle based on the two-dimensional estimated position in the priori state information of each first obstacle and the two-dimensional detected position in the target detection information of each second obstacle;

[0019] Calculating a residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0020] determining a cost matrix according to a weighted average of the overlap degree and the residual value between each first obstacle and each second obstacle;

[0021] Determining a matching relationship between the first obstacle and the second obstacle using a preset matching algorithm according to the cost matrix;

[0022] According to the matching relationship between the first obstacle and the second obstacle, a first obstacle matching each second obstacle is determined.

[0023] Optionally, calculating the residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle specifically includes:

[0024] determining a residual vector between each first obstacle and each second obstacle based on a difference between the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0025] Obtaining the residual value between each first obstacle and each second obstacle according to the square of the residual vector between each first obstacle and each second obstacle;

[0026] The three-dimensional estimated position includes the x-axis coordinate and y-axis coordinate of the first obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the first obstacle, and the heading angle of the first obstacle; the three-dimensional detected position includes the x-axis coordinate and y-axis coordinate of the second obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the second obstacle, and the heading angle of the second obstacle.

[0027] Optionally, inputting the posterior information of each first obstacle in the first image at the first moment into the priori motion model to estimate the priori state information of the first obstacle at the second moment specifically includes:

[0028] determining a priori motion model according to an environmental scene corresponding to the first image;

[0029] Determining the noise covariance of each obstacle in the first image based on statistical results of noise distribution in different environmental scenes and different obstacle categories;

[0030] Determining a priori motion covariance of each obstacle in the first image based on statistical results of acceleration noise distribution and rotation speed noise distribution of a priori motion models for different obstacle categories and different motion scenes;

[0031] Priori state information of the first obstacle at the second moment is estimated based on the priori motion model, the noise covariance, the priori motion covariance, and the posterior information of the first obstacle at the first moment.

[0032] Optionally, when the second moment is a moment corresponding to a first frame of image captured by the camera after the vehicle is started, the method further includes:

[0033] Preset parameters are obtained, and the preset parameters are used as prior state information of each first obstacle at the second moment.

[0034] In a second aspect, the present application provides an obstacle detection device, comprising:

[0035] an acquisition module configured to input a priori information of each first obstacle in a first image at a first moment into a priori motion model, estimate priori state information of the first obstacle at a second moment, the priori state information including an estimated position of the first obstacle at the second moment, and the a priori information including a predicted position of the first obstacle at the first moment, the first moment and the second moment being adjacent previous moments; and identify, based on a second image captured by a camera on the vehicle at the second moment, obtain target detection information of a second obstacle in the second image at the second moment, the target detection information including a detected position of the second obstacle;

[0036] a processing module configured to determine, based on the prior state information of the first obstacle and the target detection information of the second obstacle, a first obstacle that matches each second obstacle; and, based on a preset filtering algorithm, fuse the target detection information of the second obstacle with the prior state information of the first obstacle that matches the second obstacle to obtain the posterior information of the second obstacle.

[0037] Optionally, the acquisition module is specifically configured to:

[0038] identifying the second obstacles in the second image using a two-dimensional recognition algorithm, and obtaining a two-dimensional detection position of each second obstacle, wherein the two-dimensional detection position includes a two-dimensional positioning coordinate, a two-dimensional width, and a two-dimensional length of a two-dimensional detection frame of the second obstacle;

[0039] determining a first distance in the two-dimensional detection position according to the two-dimensional positioning coordinates, the two-dimensional width, the two-dimensional length, and camera information of the camera on the vehicle in the two-dimensional detection position;

[0040] identifying the second obstacles in the second image using a three-dimensional recognition algorithm to obtain a three-dimensional detected position of each second obstacle, the three-dimensional detected position including a center point coordinate, a three-dimensional width, a three-dimensional length, and a three-dimensional height of the second obstacle in vehicle coordinates;

[0041] A second distance in the three-dimensional detection position is determined based on the center point coordinates, the three-dimensional width, the three-dimensional length, and the three-dimensional height in the three-dimensional detection position.

[0042] Optionally, the processing module is specifically configured to:

[0043] Calculating a degree of overlap between each first obstacle and each second obstacle based on the two-dimensional estimated position in the priori state information of each first obstacle and the two-dimensional detected position in the target detection information of each second obstacle;

[0044] Calculating a residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0045] determining a cost matrix according to a weighted average of the overlap degree and the residual value between each first obstacle and each second obstacle;

[0046] Determining a matching relationship between the first obstacle and the second obstacle using a preset matching algorithm according to the cost matrix;

[0047] According to the matching relationship between the first obstacle and the second obstacle, a first obstacle matching each second obstacle is determined.

[0048] Optionally, the processing module is specifically configured to:

[0049] determining a residual vector between each first obstacle and each second obstacle based on a difference between the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0050] Obtaining the residual value between each first obstacle and each second obstacle according to the square of the residual vector between each first obstacle and each second obstacle;

[0051] The three-dimensional estimated position includes the x-axis coordinate and y-axis coordinate of the first obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the first obstacle, and the heading angle of the first obstacle; the three-dimensional detected position includes the x-axis coordinate and y-axis coordinate of the second obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the second obstacle, and the heading angle of the second obstacle.

[0052] Optionally, the acquisition module is specifically configured to:

[0053] determining a priori motion model according to an environmental scene corresponding to the first image;

[0054] Determining the noise covariance of each obstacle in the first image based on statistical results of noise distribution in different environmental scenes and different obstacle categories;

[0055] Determining a priori motion covariance of each obstacle in the first image based on statistical results of acceleration noise distribution and rotation speed noise distribution of a priori motion models for different obstacle categories and different motion scenes;

[0056] Priori state information of the first obstacle at the second moment is estimated based on the priori motion model, the noise covariance, the priori motion covariance, and the posterior information of the first obstacle at the first moment.

[0057] Optionally, the acquisition module is further configured to:

[0058] When the second moment corresponds to the first frame of image captured by the camera after the vehicle is started, preset parameters are obtained and used as prior state information of each first obstacle at the second moment. In a third aspect, the present application provides a controller comprising: a processor and a memory communicatively connected to the processor. The memory stores a computer program. The processor executes the computer program stored in the memory to implement the method of the first aspect and any possible design of the first aspect.

[0059] In a fourth aspect, the present application provides a vehicle comprising: at least one camera disposed on the front side of the vehicle and a controller as in the third aspect and any possible design of the third aspect.

[0060] In a fifth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the method in the first aspect and any possible design of the first aspect.

[0061] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method in the first aspect and any possible design of the first aspect.

[0062] The obstacle detection method, controller, and vehicle provided herein input a priori state information of a first obstacle at a first moment into a priori motion model to estimate the priori state information of the first obstacle at a second moment. The priori state information indicates the estimated position of the first obstacle at the second moment. A preset obstacle recognition algorithm is used to detect second obstacles in a second image at the second moment and obtain target detection information for each second obstacle. A matching relationship between the first and second obstacles is determined based on the degree of matching between the priori state information and the target detection information. The matching degree can be determined by calculating a two-dimensional overlap and a three-dimensional residual value. A preset filtering algorithm is used to fuse the target detection information of the second obstacle with the priori state information of the first obstacle that matches the second obstacle to obtain the posterior information of the second obstacle. The posterior information includes the predicted position of the second obstacle. The preset filtering algorithm can be a Bayesian filtering algorithm to improve the accuracy of obstacle information estimation. In addition, this application fully considers the multi-dimensional observation information of two-dimensional obstacle information and three-dimensional obstacle information, improving the success rate and accuracy of association matching; and, through Bayesian filter fusion, the estimation accuracy of posterior obstacle information is improved; and, because both two-dimensional obstacle information and three-dimensional obstacle information are information of obstacle dimensions, the implementation also shortens the overall algorithm time and reduces the difficulty of deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] FIG1 is a schematic diagram of an autonomous driving scenario provided by an embodiment of the present application;

[0065] FIG2 is a flow chart of an obstacle detection method provided in one embodiment of the present application;

[0066] FIG3 is a flowchart of obtaining prior obstacle information provided by an embodiment of the present application;

[0067] FIG4 is a schematic diagram of triangulation distance measurement provided by an embodiment of the present application;

[0068] FIG5 is a schematic diagram of a size measurement method provided by an embodiment of the present application;

[0069] FIG6 is a flowchart of a matching process provided by an embodiment of the present application;

[0070] FIG7 is an example diagram of an obstacle detection method provided by an embodiment of the present application;

[0071] FIG8 is a schematic structural diagram of an obstacle detection device provided in one embodiment of the present application;

[0072] FIG9 is a schematic diagram of the hardware structure of a controller provided in an embodiment of the present application.

[0073] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0075] With the advancement of technology, intelligent driving technology is increasingly being applied to vehicles. Detecting obstacles such as vehicles, pedestrians, and objects around the vehicle is a crucial step in the intelligent driving process. The vehicle may be equipped with at least one camera. A controller inside the vehicle can detect obstacles by identifying images uploaded by the camera. In existing technologies, obstacle detection can be performed based on two-dimensional images. Detection methods include 2D and 3D detection. 2D detection can be achieved through view geometry. View geometry can include triangulation, which assumes obstacles are on the ground plane, and dimensional ranging, which assumes the obstacle's dimensions are known. Currently, methods based on view geometry have strong prior assumptions when measuring distance, and triangulation based on the horizon can also result in large errors at long distances. 3D detection based on deep learning offers higher accuracy than 2D detection. However, because 3D detection is based on single-frame detection, it can experience significant jitter at medium and long distances. As can be seen, existing ranging methods have their advantages and disadvantages. Based on these existing methods, this application proposes a detection method based on multi-observation fusion of visual observations. This method can integrate the advantages of these ranging methods to achieve better ranging performance, improve detection accuracy, and reduce detection errors.

[0076] The obstacle detection method constructed in the present application integrates 2D information and 3D information in the image space, realizes obstacle detection with multi-observation fusion, and improves detection accuracy. The present application integrates three ranging methods in view geometry: ground plane triangulation hypothesis ranging, size ranging, and 3D detection ranging, and fuses 2D image space information with 3D image space information, thereby realizing the diversity of observation sources and improving the robustness of performance. Compared with the view geometry ranging scheme based only on 2D detection, the present application has greatly improved the accuracy of long-distance ranging, the response of posterior estimation ranging and speed measurement. Compared with the ranging scheme based only on 3D detection, the present application has greatly improved the jitter of ranging and close-range ranging. Specifically, the present application statistics the noise distribution of 2D and 3D multi-dimensional observations at different distances according to scenes such as highways, ramps, tunnels and urban areas, thereby obtaining the noise covariance of each observation. Based on Bayesian fusion, the present application uses the obstacle information of the previous frame as a parameter for obstacle detection in the subsequent frame, which greatly improves the performance of obstacle detection. Moreover, the jitter of the obstacle distance and speed detected by the present application is small, the stability is high, and the response is good. Specifically, the present application associates the obstacles of the previous and next frames to ensure the stability of the obstacles in time. In addition, the present application considers the association between 2D and 3D space at the same time, which greatly solves the pain points of obstacle occlusion and long-distance association in the association. Among them, in 2D space, the present application considers the overlap (Intersection over Union, IOU) of the 2D Bounding Box, and in 3D space, the present application considers the residual value of (x, y, W, H, θ) in the 3D Bounding Box. The present application uses the weighted average of the overlap and residual value as the value in the cost matrix (Cost Matrix), and realizes the obstacle association of the previous and next frames through Hungarian matching. The association of this solution can greatly solve the problem of obstacle occlusion based on 2D image association and the problem of inaccurate long-distance association.

[0077] The following describes exemplary application scenarios of the embodiments of the present application.

[0078] Figure 1 illustrates a schematic diagram of an autonomous driving scenario provided by an embodiment of the present application. As shown in Figure 1 , multiple vehicles may be traveling on a road. Figure 1 illustrates two vehicles. To facilitate differentiation between the two vehicles, they may be named the front vehicle and the rear vehicle, respectively. The vehicles may be equipped with a controller and multiple sensors. Optionally, the sensor may be a camera. Optionally, the camera may be located on the front side of the vehicle. For example, the camera may be located in front of the vehicle's rearview mirror. The camera may be used to capture images in front of the vehicle. For example, the front vehicle may capture the zebra crossing in front of the vehicle, as well as information such as pedestrians and vehicles on the crossing. The rear vehicle may capture the rear of the front vehicle. The controller may capture the images captured by the camera, process the images, and detect obstacle information about obstacles in the images. Optionally, the obstacles may be objects other than the vehicle itself. For example, the obstacles may be pedestrians, vehicles, guardrails, landscaping, traffic lights, road signs, etc. Optionally, the obstacle information may include information such as the distance between the obstacle and the vehicle itself, the size of the obstacle, the speed of the obstacle, and the heading angle of the obstacle.

[0079] In this application, a controller is used as the execution entity to execute the obstacle detection method of the following embodiments. Specifically, the execution entity can be a hardware device of the controller, or a software application in the controller that implements the following embodiments, or a computer-readable storage medium that has installed thereon the software application that implements the following embodiments, or the code that implements the software application of the following embodiments.

[0080] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0081] FIG2 shows a flow chart of an obstacle detection method provided by an embodiment of the present application. Based on the embodiment shown in FIG1 , as shown in FIG2 , with the controller as the execution body, the method of this embodiment may include the following steps:

[0082] S101: Input a priori information of each first obstacle in the first image at a first moment into a priori motion model to estimate priori state information of the first obstacle at a second moment, where the priori state information includes an estimated position of the first obstacle at the second moment, and the a priori information includes a predicted position of the first obstacle at the first moment, where the first moment and the second moment are adjacent previous moments.

[0083] In this embodiment, the controller may obtain a priori information about each first obstacle in the first image at a first moment. The controller may input the a priori information about the first obstacle into a priori motion model. The priori motion model is used to estimate the motion of the first obstacle. The controller may use the priori motion model to estimate a priori state information about the first obstacle at a second moment. The priori state information is used to indicate the estimated position of the first obstacle at the second moment.

[0084] The first moment is a moment set based on the second moment. The second moment is the moment currently being calculated. The first moment is the moment immediately preceding the second moment. For ease of distinction, in this embodiment, the image captured at the first moment is named the first image. The obstacle in the first image is named the first obstacle. The image captured at the second moment is named the second image. The obstacle in the second image is named the second obstacle. The names of the second obstacle and the first obstacle are used only to distinguish the images in which the obstacle exists and are not used to distinguish the type of obstacle.

[0085] In which, the vehicle may be provided with a camera. Optionally, the camera may be used to capture images in front of the vehicle. Optionally, the camera may periodically capture images and upload them to the vehicle's controller. Optionally, the period may be determined based on the camera's capture rate. Alternatively, the period may be set by a technician based on experience. Alternatively, the period may be determined based on current road conditions. Alternatively, the camera may capture a video and upload it to the controller. The controller may obtain image frames from the video based on a preset time interval. Optionally, the preset time interval may be determined based on the frame rate of the video. Alternatively, the preset time interval may be set by a technician based on experience. Alternatively, the preset time interval may be determined based on current road conditions. Optionally, the first moment and the second moment may correspond to two adjacent periods.

[0086] The a posteriori information of the first obstacle is the final predicted position information of the first obstacle at the first moment. The use of this a posteriori information of the first obstacle at the first moment enables the association of this obstacle information between the preceding and succeeding frames. In terms of time sequence, the final a posteriori information of the obstacle calculated at the previous moment is used as a known parameter to estimate the a priori state information of the obstacle at the second moment. For example, in this embodiment, after step S104 completes the calculation of the a posteriori information of the second obstacle at the second moment, the controller can use this second obstacle's a posteriori information when calculating the a posteriori information of the third obstacle at the third moment to estimate the a priori state information of the second obstacle at the third moment.

[0087] In one example, the process of the controller calculating the prior state information using the prior motion model may include the following steps:

[0088] Step 1: Determine a priori motion model based on the environment scene corresponding to the first image;

[0089] In this step, the controller can select different prior motion models according to different scenarios. The prior motion models can be constant velocity models (CV), constant turn rate and velocity models (CTRV), etc.

[0090] Step 2: Determine the noise covariance of each obstacle in the first image based on the statistical results of noise distribution in different environmental scenes and different obstacle categories;

[0091] In this step, the controller can also calculate the noise distribution of each observation for different scenes, such as highways, urban areas, ramps, and zebra crossings, as well as for different obstacle categories. Based on the first image, the controller can determine the scene and obstacle category corresponding to the first image. Based on the statistical results and the scene and obstacle category corresponding to the first image, the controller can dynamically and automatically generate noise covariance.

[0092] Step 3: Determine the prior motion covariance of each obstacle in the first image based on the statistical results of the acceleration noise distribution and the rotational speed noise distribution of the prior motion model for different obstacle categories and different motion scenes;

[0093] In this step, the controller can calculate the acceleration noise distribution and the rotational speed noise distribution of the prior motion model based on different obstacle types and different motion scenarios, such as whether the obstacle is moving straight or crossing. Based on the statistical results, the controller can set a corresponding prior motion covariance for each first obstacle in the first image.

[0094] Step 4: Estimate the prior state information of the first obstacle at the second moment based on the prior motion model, the noise covariance, the prior motion covariance, and the posterior information of the first obstacle at the first moment.

[0095] In this step, the controller can determine the predicted position of the first obstacle at the first moment based on the a priori information about the first obstacle. The server can use the a priori motion model to estimate the estimated position that the first obstacle can reach from the predicted position from the first moment to the second moment. The estimated position that the first obstacle can reach at the second moment is the a priori state information of the first obstacle at the second moment.

[0096] Optionally, to improve the accuracy of the estimation, the controller may add noise covariance to the estimation process of the prior motion model. The addition of the noise covariance simulates the possible motion of the first obstacle in the scene corresponding to the first image.

[0097] Optionally, to improve the accuracy of the estimation, the controller may further add a priori motion covariance to the estimation process of the priori motion model. The addition of the priori motion covariance simulates the possible motion of the first obstacle under its original motion state.

[0098] S102 : Identify and obtain target detection information of a second obstacle in the second image at the second moment based on a second image captured by a camera on the vehicle at the second moment, where the target detection information includes a detected position of the second obstacle.

[0099] In this embodiment, the controller may use a second image captured by a camera on the vehicle at the second moment. It should be noted that in this embodiment, the actual calculation time is the second moment, and the posterior information of the second obstacle, that is, the predicted position of the second obstacle.

[0100] The controller may be pre-installed with an obstacle recognition algorithm. This obstacle recognition algorithm can be used to detect obstacles in an image and obtain a detection frame for the obstacle. Optionally, the obstacle recognition algorithm may include at least a two-dimensional obstacle recognition algorithm and a three-dimensional obstacle recognition algorithm. The two-dimensional obstacle recognition algorithm may be a 2D object detection algorithm, such as CNN, SSD, or YOLO. The three-dimensional obstacle recognition algorithm may be a 3D object detection algorithm, such as a pure image monocular 3D detection algorithm.

[0101] Optionally, each 2D object detection algorithm and 3D object detection algorithm can be used to detect a type of target. For example, the 2D object detection algorithm and 3D object detection algorithm can be used to detect one type of target, such as a vehicle, a pedestrian, a bicycle, a roadblock, a green belt, or a road sign. In this case, the obstacle recognition algorithm can include multiple 2D object detection algorithms and / or multiple 3D object detection algorithms.

[0102] Alternatively, the 2D object detection algorithm and 3D object detection algorithm included in the obstacle recognition algorithm can be used to detect multiple types of targets. For example, the target detection algorithm can simultaneously detect vehicles, pedestrians, bicycles, roadblocks, green belts, road signs, and other targets. In this case, the obstacle recognition algorithm can include a 2D object detection algorithm and a 3D object detection algorithm.

[0103] The controller may use a preset obstacle recognition algorithm to detect second obstacles in the second image and obtain target detection information for each second obstacle. The target detection information may include a detection location. Specifically, the detection location may be a detection frame. Optionally, the obstacle recognition algorithm includes a two-dimensional obstacle detection algorithm and a three-dimensional obstacle detection algorithm. Therefore, the detection location may include a two-dimensional detection location and a three-dimensional detection location.

[0104] In addition, the controller can also calculate a first distance between the obstacle and the vehicle based on the two-dimensional detection information. Furthermore, the controller can directly calculate a second distance between the obstacle and the vehicle based on the three-dimensional obstacle information. The target detection information can include the first and second distances.

[0105] The controller can output a two-dimensional detected position of the second obstacle using a two-dimensional obstacle detection algorithm. The two-dimensional detected position can include a two-dimensional detection frame. The two-dimensional detected position can include information such as the two-dimensional positioning coordinates, two-dimensional width, and two-dimensional length of the two-dimensional detection frame.

[0106] The controller can also calculate the first distance between the obstacle and the ego vehicle based on the two-dimensional detection information and triangulation based on the ground plane assumption of the view geometry. Alternatively, the controller can also measure the distance to the obstacle based on the two-dimensional obstacle information and based on a size assumption to obtain the first distance between the obstacle and the ego vehicle. Alternatively, the controller can triangulate the distance based on the ground plane assumption of the view geometry and the size assumption to obtain two distances between the obstacle and the ego vehicle, and then take a weighted average of the two distances as the first distance. Optionally, the weighting can be 1:1.

[0107] The controller can output the 3D detected position of the second obstacle using a 3D obstacle detection algorithm. This 3D detected position can include a 3D detection frame. The 3D detected position includes the center coordinates, 3D width, 3D length, 3D height, and heading angle of the second obstacle in vehicle coordinates. The controller can also directly obtain the second distance between the obstacle and the vehicle and the obstacle's size based on the 3D obstacle information.

[0108] S103: Determine a first obstacle that matches each second obstacle based on the priori state information of the first obstacle and the target detection information of the second obstacle.

[0109] In this embodiment, the controller can obtain the prior state information of each first obstacle according to step S101. The controller can also obtain the target detection information of each second obstacle according to step S102. Since the first moment and the second moment are two adjacent moments, there is a certain overlap between the first obstacle identified in the first image at the first moment and the second obstacle identified in the second image at the second moment. That is, the first obstacle in the first image and the second obstacle in the second image may correspond to the same obstacle. On the basis that the default prior motion model is reliable, the prior state information and target detection information of the same obstacle should have a high overlap. Based on this idea, the controller determines the matching relationship between the first obstacle and the second obstacle by calculating the matching degree of the prior state information and the target detection information. Optionally, the matching degree can be determined by calculating the two-dimensional overlap and the three-dimensional residual value.

[0110] In outdoor scenes, the types and numbers of obstacles often vary significantly. In busy sections of road, the first and second images may include multiple obstacles, with multiple obstacles of each type. On quieter sections of road, however, there may not even be any obstacles in the first or second images. Based on this, the controller considers the most complex scenarios and uses a multi-obstacle matching method to match the first obstacle in the first image with the second obstacle in the second image.

[0111] When associating multiple obstacles, the controller makes full use of the two-dimensional estimated position and three-dimensional estimated position in the prior state information of the first obstacle, and associates it with the two-dimensional detection position and three-dimensional detection position in the target detection information. Specifically, the controller can calculate the overlap of the two-dimensional estimated position and the two-dimensional detection position. The controller can also calculate the residual value of the three-dimensional estimated position and the three-dimensional detection position. The server can use the weighted average of the overlap and the residual value as the value in the cost matrix (Cost Matrix) of Hungarian matching. The controller can determine the matching pairs of multiple second obstacles and the first obstacle through Hungarian matching. When the second obstacle matches the first obstacle, it means that the second obstacle and the first obstacle are the same obstacle. Compared with the existing matching method, this matching method that integrates two-dimensional and three-dimensional matching has greatly improved the matching rate and matching accuracy. In addition, the matching method that integrates two-dimensional and three-dimensional matching greatly solves the problems of obstacle occlusion and inaccurate long-distance matching when matching is based only on 2D images.

[0112] Optionally, if the current moment is the second moment, and the image captured at the current moment is the first image captured after the vehicle was started, or the image captured at the current moment is the first frame of a video uploaded from a camera, the controller has not yet calculated the a posteriori information of the first obstacle at the first moment (the moment before the current moment). In this case, the controller can obtain preset parameters and use them as the a priori state information of the first obstacle at the second moment. Optionally, the preset parameters can be initial parameters set by a technician based on experience.

[0113] S104 : Based on a preset filtering algorithm, fuse the target detection information of the second obstacle and the priori state information of the first obstacle that matches the second obstacle to obtain the posterior information of the second obstacle.

[0114] In this embodiment, the controller can determine the first obstacle corresponding to each second obstacle based on the matching relationship determined in step S103. That is, the second obstacle and the first obstacle correspond to the same obstacle. The controller can obtain prior state information of the first obstacle in step S101. The controller can obtain target detection information of the second obstacle in step S102. The controller can fuse the prior state information and target detection information using a preset filtering algorithm to obtain a posteriori information of the second obstacle. This a posteriori information includes the predicted position of the second obstacle. Optionally, the preset filtering algorithm can be a Bayesian filtering algorithm.

[0115] Optionally, the posterior information of the second obstacle may include at least the distance from the second obstacle to the ego vehicle. The controller may obtain the estimated first distance and second distance from the prior state information of the second obstacle. The controller may also obtain the first distance and second distance from the target detection information of the second obstacle. The controller may use the two first distances and second distances as two observations and fuse them based on Bayesian filtering to obtain the optimal predicted distance. The predicted distance is the distance from the second obstacle to the ego vehicle in the posterior information. Alternatively, the controller may use the two-dimensional predicted position and the three-dimensional predicted position in the fused posterior information to calculate the first distance and the second distance respectively.

[0116] Optionally, the controller can also perform three-dimensional modeling of the second obstacle. In order to distinguish the first distance from the second distance, this embodiment can record the longitudinal distance from the second obstacle to the vehicle in the vehicle coordinate system as x1, which is obtained by the triangulation method based on the ground plane assumption. This embodiment can also record the longitudinal distance from the second obstacle to the vehicle in the vehicle coordinate system as x2, which is obtained based on the size assumption. This embodiment can also record the longitudinal distance from the second obstacle to the vehicle in the vehicle coordinate system as x3, and the lateral distance as y3. The height and width of the second obstacle are recorded as H3 and W3 respectively. The heading angle of the second obstacle is recorded as θ3. The controller can use (x, y, vx, vy, θ3, H, W) as the state quantity of the second obstacle.

[0117] In the obstacle detection method provided herein, a controller may input a priori information about a first obstacle at a first moment into a priori motion model to estimate a priori state information of the first obstacle at a second moment. This priori state information indicates the estimated position of the first obstacle at the second moment. The controller may use a preset obstacle recognition algorithm to detect second obstacles in a second image at the second moment and obtain target detection information for each second obstacle. The controller may determine the matching relationship between the first and second obstacles based on the degree of match between the priori state information and the target detection information. This degree of match may be determined by calculating two-dimensional overlap and three-dimensional residual values. The controller may use a preset filtering algorithm to fuse the target detection information of the second obstacle with the a priori state information of the first obstacle that matches the second obstacle to obtain a priori information about the second obstacle. This a priori information includes the predicted position of the second obstacle. The preset filtering algorithm may be a Bayesian filtering algorithm. In this application, the multi-dimensional observation information of two-dimensional and three-dimensional obstacle information is fully considered to improve the success rate and accuracy of the associated matching. Furthermore, through Bayesian filtering fusion, the accuracy of the estimation of the a priori obstacle information is improved. Furthermore, since both two-dimensional and three-dimensional obstacle information are information about the obstacle dimension, the implementation of the method in this application also shortens the overall algorithm time consumption and reduces the difficulty of deployment. Therefore, the method in this application achieves dual optimization and improvement of performance and time efficiency in obstacle information estimation.

[0118] Based on the above embodiment, as shown in FIG3 , with the controller as the execution body, the specific steps of detecting the target information of the second obstacle in step S102 may include:

[0119] S201: Identify second obstacles in a second image using a two-dimensional recognition algorithm to obtain a two-dimensional detection position of each second obstacle, where the two-dimensional detection position includes a two-dimensional positioning coordinate, a two-dimensional width, and a two-dimensional length of a two-dimensional detection frame of the second obstacle.

[0120] In this embodiment, the controller can identify the second obstacles in the second image through a two-dimensional obstacle recognition algorithm and obtain the two-dimensional detection position of each second obstacle. The two-dimensional detection position may include a two-dimensional detection box (Bounding Box). The two-dimensional detection position may include information such as the two-dimensional positioning coordinates, two-dimensional width and two-dimensional length of the two-dimensional detection box. Optionally, the two-dimensional detection position may include (u, v, h, w). Among them, u, v represent the two-dimensional positioning coordinates of the two-dimensional detection box of the second obstacle. Optionally, the two-dimensional positioning coordinates may be the upper left corner coordinates of the two-dimensional detection box. Alternatively, the two-dimensional positioning coordinates may be the coordinate points of the upper left corner coordinates, upper right corner coordinates, lower left corner coordinates, lower right corner coordinates, center point coordinates, etc. of the two-dimensional detection box with identification positions. The coordinate system where the positioning coordinates are located may be with the upper left corner of the second image as the coordinate origin. h, w represent the two-dimensional height and two-dimensional width of the two-dimensional detection box of the obstacle.

[0121] S202: Determine a first distance in the two-dimensional detection position according to the two-dimensional positioning coordinates, the two-dimensional width, the two-dimensional length, and the camera information of the camera on the vehicle in the two-dimensional detection position.

[0122] In one example, after obtaining the two-dimensional detection position of the second obstacle, the controller can use the two-dimensional detection position to measure the distance through the triangulation method. The distance measurement diagram of the triangulation method can be shown in Figure 4. Among them, the position of the right angle on the left side of the triangle is the position of the vehicle. The second obstacle and the vehicle are on the same ground plane. The position of the second obstacle is the position of the acute angle on the right side of the triangle. The second obstacle can be explained by taking the vehicle as an example. The position of the acute angle is the position of the ground contact point of the rear wheel of the second obstacle. The controller can determine the first distance between the second obstacle and the vehicle through the right triangle as shown in the figure. The specific steps may include:

[0123] Step 1: Determine a first ratio based on the ratio of the camera focal length in the camera information to the vanishing point distance of each second obstacle. The vanishing point distance is the distance from the projection of the center point of the rear wheel contact of the second obstacle on the second image to the vanishing point of the second image.

[0124] In this step, the controller can obtain the camera focal length from the camera information, which can be represented as f. The controller can also obtain the vanishing point distance of the second obstacle. The vanishing point distance of the second obstacle is the distance from the contact center point of the two rear wheels of the second obstacle in the second image to the vanishing point in the second image. This vanishing point distance can be denoted as y_cam. The use of the camera focal length and vanishing point distance in the triangulation method can be shown as f and y_cam in Figure 4. The controller can determine a first ratio based on the camera focal length and vanishing point distance.

[0125] Step 2: Determine the first distance of each second obstacle based on the product of the camera height in the camera information and the first ratio of each second obstacle.

[0126] In this step, the controller can also obtain the camera height in the camera information. The camera height can be recorded as H_cam. The first distance between the second obstacle and the vehicle can be recorded as x. As shown in Figure 4, the right triangle formed by the camera height and the first distance is the same shape as the right triangle formed by the camera focal length and the vanishing point distance. Therefore, the control can determine that the ratio of the camera height and the first distance is the same as the ratio of the camera focal length and the vanishing point distance. Therefore, the controller can determine the first distance based on the product of the camera height and the first ratio. The formula can be recorded as:

[0127] In another example, after obtaining the two-dimensional detection position of the second obstacle, the controller can use the two-dimensional detection position to measure the distance through the size ranging method. The distance measurement diagram of the size ranging method can be shown in Figure 5. As shown in Figure 5, it includes a coordinate system. Among them, the second obstacle is located in the second quadrant. The ego vehicle is located in the second quadrant and the third quadrant. The thick solid line at the location of the ego vehicle illustrates the projection of the content captured by the camera. That is, the thick solid line can be understood as the second image. The second obstacle can be explained by taking the vehicle as an example. The controller can determine the second distance between the second obstacle and the ego vehicle through the projection relationship shown in Figure 5. The specific steps may include:

[0128] Step 1: Determine the first vehicle body width of each second obstacle based on the two-dimensional detected position of each second obstacle.

[0129] In this step, the controller can determine the first vehicle width of the second obstacle based on the two-dimensional width of the two-dimensional detection frame of the second obstacle. The first vehicle width can be understood as the width of the vehicle projected into the second image. The second vehicle width can be represented by w. As shown in Figure 5, w represents the first vehicle width of the second obstacle after being projected into the second image.

[0130] Step 2: Determine the second vehicle body width of each second obstacle based on the a posteriori information of the first obstacle that matches the second obstacle.

[0131] In this step, the controller can determine the second body width of the second obstacle through the a posteriori information of the first obstacle that matches the second obstacle. The second body width is the actual width of the second obstacle. The second body width can be recorded as W. Optionally, when the second image is the first image acquired by the camera, the controller cannot obtain the second body width through the a posteriori information of the first obstacle. At this time, the controller can identify the category of the second obstacle and determine the second body width based on preset parameters and the category of the second obstacle. For example, when the category of the second obstacle is car, the second body width can be set to 1.7 meters. For another example, when the category of the second obstacle is truck, the second body width can be set to 2.5 meters.

[0132] Step 3: The controller may determine a second ratio according to a ratio of the first vehicle body width to the second vehicle body width of each second obstacle.

[0133] Step 4: Determine the first distance of each second obstacle based on the product of the camera focal length in the camera information and the second ratio.

[0134] In this step, the controller can also obtain the camera focal length from the camera information, which can be expressed as f. The camera focal length is the distance from the vehicle to the y-axis of the coordinate system. The first distance between the second obstacle and the vehicle can be recorded as X. The first distance is the distance from the second obstacle to the coordinate system y. As shown in Figure 5, the second obstacle can be projected onto the second image captured by the camera of the vehicle. Therefore, the ratio of the second body width of the second obstacle to the first body width is the same as the ratio of the second distance to the camera focal length. Therefore, the controller can determine the first distance based on the product of the camera focal length and the second ratio. The formula can be recorded as:

[0135] In another example, the first distance may be a weighted sum of the distances calculated in the above two embodiments. Optionally, the weight may be 1:1.

[0136] S203. Identify second obstacles in the second image using a three-dimensional recognition algorithm to obtain a three-dimensional detected position of each second obstacle, where the three-dimensional detected position includes the center point coordinates, three-dimensional width, three-dimensional length, and three-dimensional height of the second obstacle in vehicle coordinates.

[0137] In this embodiment, the controller can identify the second obstacle in the second image through a three-dimensional obstacle recognition algorithm and obtain the three-dimensional detection position of each second obstacle. The three-dimensional detection position may include a three-dimensional detection box (Bounding Box). The three-dimensional detection position may include (x, y, z, L, H, W, θ). Among them, x, y, z represent the center point coordinates of the three-dimensional detection box of the second obstacle. The x, y, z represent the longitudinal distance, lateral distance and elevation information in the vehicle body coordinates respectively. L, H, W represent the three-dimensional length, three-dimensional height, and three-dimensional width of the three-dimensional detection box of the obstacle in the vehicle body coordinates. θ represents the heading angle of the obstacle in the vehicle body coordinates.

[0138] S204 : Determine a second distance in the three-dimensional detection position according to the center point coordinates, the three-dimensional width, the three-dimensional length, and the three-dimensional height in the three-dimensional detection position.

[0139] In this embodiment, the controller can determine a second distance between the first obstacle and the vehicle based on the longitudinal distance, lateral distance, and elevation information of the center of the first obstacle in vehicle body coordinates. This second distance is the longitudinal distance x between the center point coordinates of the three-dimensional detected position in vehicle body coordinates.

[0140] In the obstacle detection method provided in this application, a controller can identify second obstacles in a second image using a two-dimensional obstacle recognition algorithm and obtain the two-dimensional detection position of each second obstacle. After obtaining the two-dimensional detection position of the second obstacle, the controller can determine the first distance in the two-dimensional detection position based on the two-dimensional positioning coordinates, two-dimensional width, two-dimensional length, and camera information of the vehicle's camera. The controller can identify second obstacles in the second image using a three-dimensional obstacle recognition algorithm and obtain the three-dimensional detection position of each second obstacle. The controller can determine the second distance between the first obstacle and the vehicle based on the longitudinal distance, lateral distance, and elevation information of the center of the first obstacle in vehicle coordinates. In this application, by using a two-dimensional obstacle recognition algorithm and a three-dimensional obstacle recognition algorithm, the two-dimensional detection position and the three-dimensional detection position of the second obstacle in the target detection information are obtained, achieving multi-dimensional observation, improving information richness, and thereby improving the prediction accuracy of the predicted position of the second obstacle in the posterior information.

[0141] Based on the above embodiment, as shown in FIG6 , with the controller as the execution body, the matching process may specifically include the following steps:

[0142] S301 : Calculate the degree of overlap between each first obstacle and each second obstacle based on the two-dimensional estimated position in the priori state information of each first obstacle and the two-dimensional detected position in the target detection information of each second obstacle.

[0143] In this embodiment, the controller can obtain prior state information of the first obstacle at the second moment. This prior state information may include a two-dimensional estimated position. This two-dimensional estimated position is the two-dimensional detection frame. The controller can also obtain target detection information of the second obstacle at the second moment. This target detection information may include a two-dimensional detected position. This two-dimensional detected position is the two-dimensional detection frame. The controller can form an obstacle pair consisting of each second obstacle and each first obstacle. For example, when the second image includes four second obstacles and the first image includes three first obstacles, the controller can obtain 12 obstacle pairs. These 12 obstacle pairs can be displayed in the form of a 4×3 matrix. The controller can determine the overlap between the two obstacles in an obstacle pair by calculating the IOU between the two-dimensional detection frame of the second obstacle and the two-dimensional detection frame of the first obstacle. A greater overlap indicates that the second obstacle and the first obstacle in the obstacle pair are closer in position.

[0144] S302 : Calculate a residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle.

[0145] In this embodiment, the controller can obtain prior state information of the first obstacle at the second moment. The prior state information may include a three-dimensional estimated position. The three-dimensional estimated position is the three-dimensional detection frame. The controller can also obtain target detection information of the second obstacle at the second moment. The target detection information may include a three-dimensional detected position. The three-dimensional detected position is the three-dimensional detection frame. The three-dimensional detection frame may include (x, y, H, W, θ). Where x and y represent the longitudinal and lateral distances of the obstacle's center in vehicle coordinates. H and W represent the height and width of the obstacle's three-dimensional detection frame in vehicle coordinates. θ represents the heading angle of the obstacle in vehicle coordinates. The coordinates in the three-dimensional detection frame may be coordinates in vehicle coordinates. The controller can use the difference between the three-dimensional estimated position and the three-dimensional detected position as a residual vector. The controller can calculate the square of the residual vector as a residual value. The smaller the residual value, the closer the positions of the second obstacle and the first obstacle in the obstacle pair are.

[0146] In one example, the calculation process of the residual value of an obstacle pair may specifically include the following steps:

[0147] Step 1: A controller may determine a residual vector between each first obstacle and each second obstacle based on a difference between the three-dimensional estimated position in the prior state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle. The three-dimensional estimated position includes the x-axis and y-axis coordinates of the first obstacle in vehicle coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the first obstacle, and the heading angle of the first obstacle; and the three-dimensional detected position includes the x-axis and y-axis coordinates of the second obstacle in vehicle coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the second obstacle, and the heading angle of the second obstacle.

[0148] Step 2: The controller may obtain the residual value between each first obstacle and each second obstacle according to the square of the residual vector between each first obstacle and each second obstacle.

[0149] S303: Determine a cost matrix according to a weighted average of the overlap degree and the residual value between each first obstacle and each second obstacle.

[0150] In this embodiment, the controller can calculate the weighted average of the overlap and residual values ​​for each obstacle pair. Optionally, the weighting can be 1:1, 1:2, 2:1, etc. Alternatively, the weighting of the overlap and residual values ​​can be set by a technician. The size of the cost matrix can be determined based on the number of second obstacles and the number of first obstacles. For example, when the second image includes 4 second obstacles and the first image includes 3 first obstacles, the size of the cost matrix is ​​4×3. Each value in the cost matrix can correspond to the weighted average of an obstacle pair.

[0151] S304: Determine a matching relationship between the first obstacle and the second obstacle using a preset matching algorithm according to the cost matrix.

[0152] In this embodiment, the controller can use a preset matching algorithm to obtain multiple obstacle pairs with matching relationships based on the cost matrix. The controller can define these matching obstacle pairs as matching pairs. The second obstacle in each matching pair is actually the same obstacle as the first obstacle. Optionally, the preset matching algorithm can be a Hungarian matching algorithm.

[0153] S305: Determine a first obstacle that matches each second obstacle based on the matching relationship between the first obstacle and the second obstacle.

[0154] In the obstacle detection method provided in this application, a controller can determine the degree of overlap between two obstacles in an obstacle pair by calculating the IOU between the two-dimensional detection frame of the second obstacle and the two-dimensional detection frame of the first obstacle. The controller can use the difference between the three-dimensional estimated position and the three-dimensional detected position as a residual vector. The controller can calculate the square of the residual vector as the residual value. The controller can calculate the weighted average of the degree of overlap and the residual value for each obstacle pair. The controller can obtain multiple obstacle pairs with matching relationships based on the cost matrix using a preset matching algorithm. The controller can determine the first obstacle that matches each second obstacle based on the matching relationship between the first obstacle and the second obstacle. In this application, the accuracy and efficiency of matching the second obstacle with the first obstacle are improved by using 2D information and 3D information.

[0155] Based on the above embodiment, FIG7 shows an implementation of the above embodiment.

[0156] Among them, at the moment t=0, the controller obtains the first image (Image) for processing. The controller can use a preset 2D obstacle detection algorithm to detect and obtain a 2D output. The 2D output may specifically include two-dimensional detection information in the target detection information. The controller can also use a preset 3D obstacle detection algorithm to detect and obtain a 3D output. The 3D output may specifically include three-dimensional detection information in the target detection information. The controller can perform triangulation and size measurement based on the 2D output to obtain a first distance. The controller can use the first distance and the second distance determined according to the 3D output as two observations. The controller can input the two observations into a Bayesian filter to implement a Bayesian filter with multi-observation fusion. The result obtained by the Bayesian filter is the posterior information of the first obstacle in the first image at the moment t.

[0157] Among them, at time t=T, the controller can obtain the second image (Image) and perform the same calculation as at time t=0. Optionally, the time T is any time other than time 0. For example, the time T can be time t=1. The difference is that when executed at time T, the 2D output and posterior information of the previous moment can be Hungarian matched with the 2D output of time T to determine the correlation between the first obstacle and the second obstacle in the time sequence of two consecutive moments. The controller can use the information of the first obstacle associated with the second obstacle to participate in the calculation of the posterior information of the second obstacle.

[0158] FIG8 is a schematic structural diagram of an obstacle detection device provided in an embodiment of the present application. As shown in FIG8 , the obstacle detection device 10 of this embodiment is used to implement the operations corresponding to the controller in any of the above method embodiments. The obstacle detection device 10 of this embodiment includes:

[0159] an acquisition module 11 configured to input a priori information of each first obstacle in a first image at a first moment into a priori motion model, estimate priori state information of the first obstacle at a second moment, the priori state information including an estimated position of the first obstacle at the second moment, and the a priori information including a predicted position of the first obstacle at the first moment, the first moment and the second moment being adjacent previous moments; and identify, based on a second image captured by a camera on the vehicle at the second moment, obtain target detection information of a second obstacle in the second image at the second moment, the target detection information including a detected position of the second obstacle;

[0160] The processing module 12 is configured to determine, based on the prior state information of the first obstacle and the target detection information of the second obstacle, a first obstacle that matches each second obstacle; and, based on a preset filtering algorithm, fuse the target detection information of the second obstacle with the prior state information of the first obstacle that matches the second obstacle to obtain the posterior information of the second obstacle.

[0161] Optionally, the acquisition module 11 is specifically configured to:

[0162] identifying the second obstacles in the second image using a two-dimensional recognition algorithm, and obtaining a two-dimensional detection position of each second obstacle, wherein the two-dimensional detection position includes a two-dimensional positioning coordinate, a two-dimensional width, and a two-dimensional length of a two-dimensional detection frame of the second obstacle;

[0163] determining a first distance in the two-dimensional detection position according to the two-dimensional positioning coordinates, the two-dimensional width, the two-dimensional length, and camera information of the camera on the vehicle in the two-dimensional detection position;

[0164] identifying the second obstacles in the second image using a three-dimensional recognition algorithm to obtain a three-dimensional detected position of each second obstacle, the three-dimensional detected position including a center point coordinate, a three-dimensional width, a three-dimensional length, and a three-dimensional height of the second obstacle in vehicle coordinates;

[0165] A second distance in the three-dimensional detection position is determined based on the center point coordinates, the three-dimensional width, the three-dimensional length, and the three-dimensional height in the three-dimensional detection position.

[0166] Optionally, the processing module 12 is specifically configured to:

[0167] Calculating a degree of overlap between each first obstacle and each second obstacle based on the two-dimensional estimated position in the priori state information of each first obstacle and the two-dimensional detected position in the target detection information of each second obstacle;

[0168] Calculating a residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0169] determining a cost matrix according to a weighted average of the overlap degree and the residual value between each first obstacle and each second obstacle;

[0170] Determining a matching relationship between the first obstacle and the second obstacle using a preset matching algorithm according to the cost matrix;

[0171] According to the matching relationship between the first obstacle and the second obstacle, a first obstacle matching each second obstacle is determined.

[0172] Optionally, the processing module 12 is specifically configured to:

[0173] determining a residual vector between each first obstacle and each second obstacle based on a difference between the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle;

[0174] Obtaining the residual value between each first obstacle and each second obstacle according to the square of the residual vector between each first obstacle and each second obstacle;

[0175] The three-dimensional estimated position includes the x-axis coordinate and y-axis coordinate of the first obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the first obstacle, and the heading angle of the first obstacle; the three-dimensional detected position includes the x-axis coordinate and y-axis coordinate of the second obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the second obstacle, and the heading angle of the second obstacle.

[0176] Optionally, the acquisition module 11 is specifically configured to:

[0177] determining a priori motion model according to an environmental scene corresponding to the first image;

[0178] Determining the noise covariance of each obstacle in the first image based on statistical results of noise distribution in different environmental scenes and different obstacle categories;

[0179] Determining a priori motion covariance of each obstacle in the first image based on statistical results of acceleration noise distribution and rotation speed noise distribution of a priori motion models for different obstacle categories and different motion scenes;

[0180] Priori state information of the first obstacle at the second moment is estimated based on the priori motion model, the noise covariance, the priori motion covariance, and the posterior information of the first obstacle at the first moment.

[0181] Optionally, the acquisition module 11 is further configured to:

[0182] When the second moment is the moment corresponding to the first frame image captured by the camera after the vehicle is started, preset parameters are obtained and used as the priori state information of each first obstacle at the second moment.

[0183] The obstacle detection device 10 provided in the embodiment of the present application can execute the above method embodiment. Its specific implementation principles and technical effects can be found in the above method embodiment, and will not be repeated in this embodiment.

[0184] Figure 9 shows a schematic diagram of the hardware structure of a controller provided in an embodiment of the present application. As shown in Figure 9, the controller 20 is used to implement the operations corresponding to the controller in any of the above method embodiments. The controller 20 of this embodiment may include: a memory 21, a processor 22, and a communication interface 24.

[0185] Memory 21 is used to store computer programs. Memory 21 may include high-speed random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. It may also be a USB flash drive, a mobile hard drive, a read-only memory, a magnetic disk, or an optical disk.

[0186] Processor 22 is configured to execute a computer program stored in memory to implement the obstacle detection method in the above-described embodiment. For details, please refer to the relevant descriptions in the aforementioned method embodiments. Processor 22 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented using a combination of hardware and software modules within the processor.

[0187] Optionally, the memory 21 may be independent or integrated with the processor 22 .

[0188] When the memory 21 is a device independent of the processor 22, the controller 20 may further include a bus 23. The bus 23 is used to connect the memory 21 and the processor 22. The bus 23 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0189] The communication interface 24 can be connected to the processor 21 via the bus 23. The processor 22 can control the communication interface 24. The communication interface 24 can be used to implement communication between the vehicle's sensors and the controller 20.

[0190] The controller provided in this embodiment can be used to execute the above-mentioned heating control method. Its implementation method and technical effects are similar and will not be described in detail in this embodiment.

[0191] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the methods provided in the various embodiments described above.

[0192] Among them, the computer-readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a computer-readable storage medium is coupled to a processor so that the processor can read information from the computer-readable storage medium and write information to the computer-readable storage medium. Of course, the computer-readable storage medium can also be an integral part of the processor. The processor and the computer-readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the computer-readable storage medium can also exist in a communication device as discrete components.

[0193] Specifically, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0194] The present application also provides a computer program product, comprising a computer program stored in a computer-readable storage medium. At least one processor of a device can read the computer program from the computer-readable storage medium, and at least one processor executes the computer program so that the device implements the methods provided in the various embodiments described above.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0196] The modules may be physically separate, for example, installed in different locations on a single device, or installed on different devices, or distributed across multiple network units, or distributed across multiple processors. The modules may also be integrated, for example, installed in the same device, or integrated into a set of codes. The modules may exist in the form of hardware, or in the form of software, or may be implemented in the form of software plus hardware. The present application may select some or all of the modules according to actual needs to achieve the purpose of the present embodiment.

[0197] When each module is implemented as an integrated module in the form of a software function module, it can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0198] It should be understood that, although the various steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they may be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but may be performed at different times, and their execution order is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of sub-steps or stages of other steps.

[0199] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0200] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Inputting a priori information of each first obstacle in the first image at a first moment into a priori motion model to estimate priori state information of the first obstacle at a second moment, the priori state information including an estimated position of the first obstacle at the second moment, and the a priori information including a predicted position of the first obstacle at the first moment, where the first moment and the second moment are adjacent previous moments; identifying, based on a second image captured by a camera on the vehicle at a second moment, target detection information of a second obstacle in the second image at the second moment, the target detection information including a detected position of the second obstacle; determining, based on the priori state information of the first obstacle and the target detection information of the second obstacle, a first obstacle that matches each second obstacle; Based on a preset filtering algorithm, the target detection information of the second obstacle and the priori state information of the first obstacle that matches the second obstacle are fused to obtain the posterior information of the second obstacle.

2. The method according to claim 1, characterized in that The identifying and obtaining target detection information of a second obstacle in the second image at the second moment based on the second image captured by a camera on the vehicle at the second moment specifically includes: identifying the second obstacles in the second image using a two-dimensional recognition algorithm, and obtaining a two-dimensional detection position of each second obstacle, wherein the two-dimensional detection position includes a two-dimensional positioning coordinate, a two-dimensional width, and a two-dimensional length of a two-dimensional detection frame of the second obstacle; determining a first distance in the two-dimensional detection position according to the two-dimensional positioning coordinates, the two-dimensional width, the two-dimensional length, and camera information of the camera on the vehicle in the two-dimensional detection position; identifying the second obstacles in the second image using a three-dimensional recognition algorithm to obtain a three-dimensional detected position of each second obstacle, the three-dimensional detected position including a center point coordinate, a three-dimensional width, a three-dimensional length, and a three-dimensional height of the second obstacle in vehicle coordinates; A second distance in the three-dimensional detection position is determined based on the center point coordinates, the three-dimensional width, the three-dimensional length, and the three-dimensional height in the three-dimensional detection position.

3. The method according to claim 2, characterized in that The determining, based on the priori state information of the first obstacle and the target detection information of the second obstacle, the first obstacle matching each second obstacle specifically includes: Calculating a degree of overlap between each first obstacle and each second obstacle based on the two-dimensional estimated position in the priori state information of each first obstacle and the two-dimensional detected position in the target detection information of each second obstacle; Calculating a residual value between each first obstacle and each second obstacle based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle; determining a cost matrix according to a weighted average of the overlap degree and the residual value between each first obstacle and each second obstacle; Determining a matching relationship between the first obstacle and the second obstacle using a preset matching algorithm according to the cost matrix; According to the matching relationship between the first obstacle and the second obstacle, a first obstacle matching each second obstacle is determined.

4. The method according to claim 3, characterized in that The calculating, based on the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle, a residual value between each first obstacle and each second obstacle specifically includes: determining a residual vector between each first obstacle and each second obstacle based on a difference between the three-dimensional estimated position in the priori state information of each first obstacle and the three-dimensional detected position in the target detection information of each second obstacle; Obtaining the residual value between each first obstacle and each second obstacle according to the square of the residual vector between each first obstacle and each second obstacle; The three-dimensional estimated position includes the x-axis coordinate and y-axis coordinate of the first obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the first obstacle, and the heading angle of the first obstacle; the three-dimensional detected position includes the x-axis coordinate and y-axis coordinate of the second obstacle in vehicle body coordinates, the three-dimensional width and three-dimensional height of the three-dimensional detection frame of the second obstacle, and the heading angle of the second obstacle.

5. The method according to any one of claims 1 to 4, characterized in that The step of inputting the posterior information of each first obstacle in the first image at the first moment into the priori motion model to estimate the priori state information of the first obstacle at the second moment specifically includes: determining a priori motion model according to an environmental scene corresponding to the first image; Determining the noise covariance of each obstacle in the first image based on statistical results of noise distribution in different environmental scenes and different obstacle categories; Determining a priori motion covariance of each obstacle in the first image based on statistical results of acceleration noise distribution and rotation speed noise distribution of a priori motion models for different obstacle categories and different motion scenes; Priori state information of the first obstacle at the second moment is estimated based on the priori motion model, the noise covariance, the priori motion covariance, and the posterior information of the first obstacle at the first moment.

6. The method according to any one of claims 1 to 4, characterized in that When the second moment is a moment corresponding to a first frame of image captured by the camera after the vehicle is started, the method further includes: Preset parameters are obtained, and the preset parameters are used as prior state information of each first obstacle at the second moment.

7. A controller, characterized in that: The controller includes: a processor, and a memory communicatively connected to the processor; The memory stores a computer program; the processor executes the computer program stored in the memory to implement The method according to any one of claims 1 to 6.

8. A vehicle, characterized in that: The vehicle comprises: at least one camera arranged on the front side of the vehicle and a controller as shown in claim 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images