Obstacle shielding rate determination method and device, vehicle and storage medium

Abstract

The invention relates to an obstacle shielding rate determination method and device, a vehicle and a storage medium, and the method comprises the steps: detecting a dynamic obstacle through an obstacle detection model in response to the detection of the dynamic obstacle in a multi-view camera image, so as to generate a first 3D detection frame of the dynamic obstacle, detecting a static obstacle in the multi-view camera image through an obstacle grounding point detection model, determining a plurality of grounding points of the static obstacle, determining a second 3D detection frame of the static obstacle according to the plurality of grounding points, projecting the first 3D detection frame and the second 3D detection frame to an image coordinate system of the multi-view camera image, and obtaining the multi-view camera image. And generating a plurality of 2D detection frames in the image coordinate system, and determining the obstacle shielding rate of the dynamic obstacle according to the overlapping rate between the 2D detection frames in the plurality of 2D detection frames. Therefore, the detection accuracy of the visual obstacle shielding rate is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a method, device, vehicle and storage medium for determining an obstacle occlusion rate. Background Art

[0002] Obstacle detection is an important perception algorithm in the field of autonomous driving, but visual obstacle detection is usually blocked by other visual obstacles, resulting in reduced detection performance. Therefore, it is necessary not only to give the confidence score of the obstacle, but also to regulate the occlusion rate value of the downstream obstacle, and adjust the vehicle's route planning strategy by receiving the occlusion rate of the detected target. For example, in the parking perception task of autonomous driving, the accuracy of occlusion rate detection is particularly important due to complex occlusion environments such as underground parking lots and ground parking lots. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method, device, vehicle and storage medium for determining an obstacle occlusion rate.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for determining an obstacle occlusion rate is provided, comprising:

[0005] In response to detecting a dynamic obstacle in the multi-camera image, detecting the dynamic obstacle by using an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle;

[0006] Detecting static obstacles in the multi-camera image using an obstacle grounding point detection model to determine multiple grounding points of the static obstacles;

[0007] Determining a second 3D detection frame of the static obstacle according to the multiple grounding points;

[0008] Projecting the first 3D detection frame and the second 3D detection frame into an image coordinate system of the multi-camera image to generate a plurality of 2D detection frames in the image coordinate system;

[0009] An obstacle occlusion rate of the dynamic obstacle is determined according to an overlap rate between each 2D detection frame in the multiple 2D detection frames.

[0010] Optionally, determining the obstacle occlusion rate of the dynamic obstacle according to the overlap rate between each 2D detection frame in the multiple 2D detection frames includes:

[0011] Determine a target 2D detection frame of the dynamic obstacle from the multiple 2D detection frames;

[0012] Based on the target 2D detection frame, filter out a first 2D detection frame that meets a preset condition from the multiple 2D detection frames;

[0013] The obstacle occlusion rate is determined according to an overlap rate between the target 2D detection frame and the first 2D detection frame.

[0014] Optionally, determining the obstacle occlusion rate according to an overlap rate between the target 2D detection frame and the first 2D detection frame includes:

[0015] Determine a plurality of first overlapping areas between the target 2D detection frame and the first 2D detection frame;

[0016] determining a plurality of second overlapping areas between the plurality of first overlapping areas;

[0017] Determining a target overlapping area according to the plurality of first overlapping areas and the plurality of second overlapping areas;

[0018] The obstacle occlusion rate is determined according to the target overlapping area and the initial area of ​​the target 2D detection frame.

[0019] Optionally, the obstacle detection model is trained and generated in the following manner:

[0020] Obtain the initial obstacle detection data set and the laser point cloud data at the same time;

[0021] According to the laser point cloud data, annotating the obstacle 3D detection frame of the initial obstacle detection data set in the initial multi-camera image to generate an initial obstacle detection data set;

[0022] The initial obstacle detection data set is augmented by a preset image transformation rule to generate a target obstacle detection data set, wherein the preset image transformation rule includes at least one of image frame parameter transformation, image block exchange, image cropping transformation, and image block occlusion transformation;

[0023] An initial obstacle detection model is trained based on the target obstacle detection data set to generate the obstacle detection model.

[0024] Optionally, the method further comprises:

[0025] Acquire camera parameters of the initial multi-camera image, wherein the camera parameters include at least camera intrinsic parameters and camera extrinsic parameters;

[0026] According to the camera parameters, the obstacle 3D detection frame is projected into the image coordinate system of the initial multi-camera image to generate multiple 2D detection frames;

[0027] Determine a plurality of intersection-over-union ratios between each of the plurality of 2D detection frames;

[0028] When a first intersection-over-union ratio of a first 2D detection frame among the multiple 2D detection frames is greater than a set threshold, a first 3D detection frame corresponding to the first 2D detection frame is deleted from the labeled initial multi-camera image, where the first 2D detection frame is any 2D detection frame among the multiple 2D detection frames.

[0029] Optionally, determining a second 3D detection frame of the static obstacle according to the multiple grounding points includes:

[0030] Performing cluster analysis on the multiple grounding points to determine multiple grounding point sets, each grounding point in each grounding point set belonging to the same static obstacle;

[0031] Determine, according to a first grounding point set among the multiple grounding point sets, a first grounding line of a first static obstacle corresponding to the first grounding point set, wherein the first grounding point set is any grounding point set among the multiple grounding point sets;

[0032] Determining the length information and heading angle information of the first grounding wire and the obstacle category information of the first static obstacle;

[0033] Determining preset three-dimensional size information of the first static obstacle according to the obstacle category information;

[0034] A 3D detection frame of the first static obstacle is generated according to the length information, the heading angle information and the preset three-dimensional size information.

[0035] Optionally, performing cluster analysis on the multiple grounding points to determine multiple grounding point sets includes:

[0036] Determine a plurality of grounding point categories corresponding to each of the plurality of grounding points;

[0037] determining a neighborhood radius and an object quantity threshold of a first grounding point among the multiple grounding points according to a first grounding point category of the first grounding point, wherein the first grounding point is any one of the multiple grounding points;

[0038] If the number of other grounding points within the neighborhood radius of the first grounding point is greater than or equal to the object number threshold, the first grounding point is determined to be a core grounding point of the first grounding point set, and the grounding point categories of the other grounding points are all the first grounding point category;

[0039] The plurality of grounding point sets are generated according to the plurality of core grounding points.

[0040] According to a second aspect of an embodiment of the present disclosure, a device for determining an obstacle occlusion rate is provided, the device comprising:

[0041] A first generating module is configured to detect a dynamic obstacle in the multi-camera image by using an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle;

[0042] A first determination module is configured to detect static obstacles in the multi-camera image through an obstacle grounding point detection model to determine multiple grounding points of the static obstacles;

[0043] A second determination module is configured to determine a second 3D detection frame of the static obstacle according to the multiple grounding points;

[0044] a second generating module, configured to project the first 3D detection frame and the second 3D detection frame into an image coordinate system of the multi-camera image to generate a plurality of 2D detection frames in the image coordinate system;

[0045] The execution module is configured to determine the obstacle occlusion rate of the dynamic obstacle according to the overlap rate between each 2D detection frame in the multiple 2D detection frames.

[0046] According to a third aspect of an embodiment of the present disclosure, a vehicle is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to execute the executable instructions to implement the steps of the method for determining the obstacle occlusion rate described in any one of the first aspects of the present disclosure.

[0050] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the method for determining the obstacle occlusion rate described in any one of the first aspects of the present disclosure are implemented.

[0051] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] In the above manner, in response to detecting a dynamic obstacle in a multi-camera image, the dynamic obstacle is detected by an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle, the static obstacle in the multi-camera image is detected by an obstacle ground point detection model to determine multiple ground points of the static obstacle, and a second 3D detection frame of the static obstacle is determined based on the multiple ground points, and the first 3D detection frame and the second 3D detection frame are projected into the image coordinate system of the multi-camera image to generate multiple 2D detection frames in the image coordinate system, and the obstacle occlusion rate of the dynamic obstacle is determined based on the overlap rate between each 2D detection frame in the multiple 2D detection frames. Thus, information of obstacles other than the dynamic obstacle to be detected is determined, the detection frames of each obstacle are determined by fitting, the obstacle occlusion rate of the dynamic obstacle is determined based on the overlap rate between the detection frames, and the calculation is performed based on the detection frame, so that the determined visual obstacle occlusion rate is more accurate.

[0053] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0055] Figure 1 The present invention is a flowchart of a method for determining an obstacle occlusion rate according to an exemplary embodiment.

[0056] Figure 2 The present invention is a flowchart of a method for determining an obstacle occlusion rate according to an exemplary embodiment.

[0057] Figure 3 The figure is a flowchart of a method for determining a 3D detection frame according to an exemplary embodiment.

[0058] Figure 4 The invention is a block diagram of a device for determining an obstacle occlusion rate according to an exemplary embodiment.

[0059] Figure 5 is a block diagram of a vehicle 500 according to an exemplary embodiment. DETAILED DESCRIPTION

[0060] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0061] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0062] In the related art, the method for calculating the occlusion ratio of dynamic obstacles during vehicle driving includes: (1) calculating the occlusion rate based on the category of the obstacle, for example, the degree of occlusion of a human face, but it cannot be applied to other categories of data, such as: human body, other vehicles, animals, etc.; (2) calculating the occlusion rate based on multi-frame video tracking, but it cannot process the image data alone; (3) using 2D detection results to calculate the IOU (Intersection over Union) between each occlusion in the 2D detection results, but this method only considers the occlusion of the target box detected in the image, and does not consider the occlusion of other background information; (4) using 2D segmentation results to calculate the occlusion rate, but in the process of autonomous driving, the real-time performance of 2D segmentation is not high, which affects the data processing efficiency, and the accuracy of edge processing is low.

[0063] In view of this, this proposal provides a method for determining the obstacle occlusion rate. Figure 1 is a flow chart of a method for determining an obstacle occlusion rate according to an exemplary embodiment. Figure 1 As shown, the method is used in a vehicle-mounted terminal and includes the following steps.

[0064] In step S11, in response to detecting a dynamic obstacle in the multi-camera image, the dynamic obstacle is detected by an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle.

[0065] Before explaining the specific scheme of this proposal, the application scenario of this proposal will be explained first, but this explanation is an example of the application scenario of the corresponding method in this proposal and cannot constitute a limitation on the method of this proposal. This embodiment is applied to a vehicle-mounted terminal, which is arranged on an autonomous driving vehicle. The environmental information of the vehicle's current environment is detected by a variety of sensors arranged on the autonomous driving vehicle, and the detected environmental information is transmitted to the vehicle-mounted terminal, so that the vehicle-mounted terminal can make corresponding autonomous driving decisions based on the environmental information of the current environment. Among them, the various sensors may include camera sensors, infrared sensors, radar sensors, depth sensors, etc. In order to obtain environmental information at different angles around the vehicle, sensors are arranged at multiple angles of the vehicle to achieve all-round detection of environmental information within a preset range of the vehicle, thereby assisting the vehicle-mounted terminal to make more accurate autonomous driving decisions.

[0066] In the autonomous driving scenario, obstacle detection in the vehicle's current driving environment is used to plan the vehicle's autonomous driving route, and the obstacle detection results are output to the downstream module, so that the downstream module plans the vehicle's driving trajectory based on the detection results, thereby controlling the vehicle to effectively avoid obstacles. Among them, the calculation of the obstacle occlusion rate facilitates the vehicle terminal to identify the type of obstacle, and make different obstacle avoidance route plans based on different obstacle types and obstacle occlusion rates.

[0067] For example, in this embodiment, a multi-camera device is provided at different angles of the vehicle. The user collects environmental images at different angles of the vehicle, generates multi-camera images and uploads them to the vehicle terminal. The vehicle terminal first splices the multi-camera images to generate a complete panoramic image of environmental information within the preset range of the vehicle, and then detects the environmental features in the panoramic image of the environmental information. When a dynamic obstacle is detected in the current environment, the dynamic obstacle is detected by the obstacle detection model to generate a first 3D detection frame of the dynamic obstacle. Among them, when the vehicle terminal detects the panoramic image of environmental information, it can determine that there is a dynamic object in the image based on the relevant image recognition model based on the motion state of the current vehicle, and when the dynamic object moves relative to the surrounding static obstacles, it determines that the object is a dynamic obstacle. Specifically, it can determine whether the dynamic obstacle is a dynamic obstacle that needs to be detected for obstacle occlusion rate based on the relative position between the dynamic obstacle and the vehicle based on the image recognition model. If the distance between the dynamic obstacle and the vehicle is less than the set distance threshold, and the confidence threshold of the dynamic obstacle in the image recognition model is greater than the set threshold, then the dynamic obstacle is determined to be an obstacle that needs to be detected.

[0068] In this embodiment, the obstacle detection model is used to detect dynamic obstacles in multi-camera images. The detection method includes detecting the position of the dynamic obstacle and detecting the laser point cloud data of the dynamic obstacle based on the depth camera device, so as to determine the position of the dynamic obstacle in the multi-camera image and the depth of each position, and generate a minimum detection frame surrounding the dynamic obstacle based on the above detection information as the first 3D detection frame of the dynamic obstacle, and mark the first 3D detection frame of the dynamic obstacle at the relative position of the dynamic obstacle in the multi-camera image. The obstacle detection model is trained by multiple obstacle detection data sets, each of which includes different types of dynamic obstacles and corresponding 3D detection frame annotation information. The obstacle detection model is trained based on the obstacle detection data set so that the obstacle detection model can accurately identify the 3D detection frame of the dynamic obstacle in the image.

[0069] It should be noted that in this embodiment, there may be multiple dynamic obstacles detected in the vehicle's current environment through the multi-camera image at the same time. When a dynamic obstacle meets the above detection conditions in the multi-camera image, it can be determined that the dynamic obstacle is a dynamic obstacle to be detected that requires obstacle shielding rate detection. When there are multiple dynamic obstacles, different colors of first 3D detection frames can be marked for different dynamic obstacles in the multi-camera image to distinguish different dynamic obstacles.

[0070] Optionally, in one implementation, the obstacle detection model may be trained and generated in the following manner:

[0071] Obtain the initial obstacle detection data set and the laser point cloud data at the same time;

[0072] According to the laser point cloud data, the obstacle 3D detection frame of the initial obstacle detection dataset is annotated in the initial multi-camera image to generate the initial obstacle detection training set;

[0073] The initial obstacle detection training set is augmented by a preset image transformation rule to generate a target obstacle detection training set, wherein the preset image transformation rule includes at least one of image frame parameter transformation, image block exchange, image cropping transformation, and image block occlusion transformation;

[0074] An initial obstacle detection model is trained based on the target obstacle detection training set to generate an obstacle detection model.

[0075] For example, in this embodiment, an initial obstacle detection data set is collected from a database. The initial obstacle detection data set is a plurality of different types of multi-camera image data, each of which includes dynamic obstacles, and laser point cloud data corresponding to each multi-camera image data at the same time. The depth distance information between the dynamic obstacle and the vehicle in the multi-camera image can be determined through the laser point cloud data. According to the initial obstacle detection data set and the laser point cloud data, an obstacle 3D detection frame of the dynamic obstacle in the multi-camera image is generated, and the obstacle 3D detection frame is annotated on the initial obstacle detection data set to generate an initial obstacle detection training set.

[0076] In order to improve the robustness of the model and make the 3D detection frame determined by the model detection more accurate, the initial obstacle detection training set is expanded during the model training process of this embodiment. For example, in this embodiment, the initial obstacle detection training set is expanded by preset image transformation rules to generate a target obstacle detection training set. The preset image transformation rules include image frame parameter transformation. For example, the color, brightness, transparency, contrast and other image parameters of the 3D detection frame in the obstacle training set can be transformed to generate a new obstacle training set; image block exchange: the 3D detection frame in the obstacle training set is used as an image block, and the image blocks in different training sets are exchanged to generate a new obstacle training set; image cropping transformation: the 3D detection frame in the obstacle training set is cropped based on the 3D detection frame in the image to generate a new obstacle training set; image block occlusion: some blocks in the obstacle training set are coded by using mosaic coding to generate a new obstacle training set. Based on the new obstacle detection training set generated by the transformation and the initial obstacle detection training set, the target obstacle detection training set is generated.

[0077] The preset initial obstacle detection model is trained through the target obstacle detection training set, and the model parameters in the initial obstacle detection model are adjusted based on the model training results of multiple iterations. After the initial obstacle detection model converges, an obstacle detection model is generated.

[0078] Optionally, in one implementation, the above model training method further includes:

[0079] Obtaining camera parameters of an initial multi-camera image, where the camera parameters include at least camera intrinsic parameters and camera extrinsic parameters;

[0080] According to the camera parameters, the obstacle 3D detection frame is projected into the image coordinate system of the initial multi-camera image to generate multiple 2D detection frames;

[0081] Determine a plurality of intersection-over-union ratios between each of the plurality of 2D detection frames;

[0082] When a first intersection-over-union ratio of a first 2D detection frame among multiple 2D detection frames is greater than a set threshold, a first 3D detection frame corresponding to the first 2D detection frame is deleted from the labeled initial multi-camera image, where the first 2D detection frame is any 2D detection frame among the multiple 2D detection frames.

[0083] For example, in this embodiment, in order to improve the accuracy of the obstacle detection model during the training process, it is necessary to screen the 3D detection frames in the obstacle detection data set and eliminate the 3D detection frames that do not need to be detected and analyzed. In this embodiment, after the obstacle 3D detection frame is generated based on the initial obstacle detection data set and the laser point cloud data, the 3D detection frame in the data set needs to be screened. For example, in this embodiment, for a multi-eye camera image corresponding to any obstacle detection data set (a 3D detection frame with a dynamic obstacle marked in the multi-eye camera image), the camera parameters when the multi-eye camera image is generated are obtained, wherein the camera parameters at least include camera internal parameters, such as camera contrast, camera shooting depth, camera focus distance, camera saturation, etc.; and camera external parameters, such as camera setting position, camera setting angle, and the spatial range of the camera acquisition image. The 3D detection frame in the multi-eye camera image is transformed by the camera parameters, and the 3D detection frame is projected into the image coordinate system corresponding to the multi-eye camera image to generate multiple 2D detection frames. Determine the intersection-and-union ratio between each 2D detection frame and other 2D detection frames. When there is a 2D detection frame whose intersection-and-union ratio is greater than a set threshold, determine the 3D detection frame corresponding to the 2D detection frame and delete the 3D detection frame from the multi-camera image. For example, when it is determined that the intersection-and-union ratio of the target 2D detection frame is greater than 90%, the target 3D detection frame corresponding to the target 2D detection frame is deleted from the multi-camera image.

[0084] In step S12, the static obstacles in the multi-camera image are detected by using the obstacle grounding point detection model to determine multiple grounding points of the static obstacles.

[0085] For example, in this embodiment, the vehicle terminal is loaded with an obstacle grounding point detection model, which is used to detect the grounding points of static obstacles in the multi-camera image. The obstacle grounding point data set with the static obstacle grounding points can be used to train the initial obstacle grounding point detection model, wherein there are many different types of static obstacles and grounding points of static obstacles in the obstacle grounding point data set, and the static obstacles can be walls, stone pillars, bushes, flower beds, etc. The grounding lines of static obstacles are first marked, and then the important grounding points in the grounding lines are marked based on preset rules, so as to generate the obstacle grounding point data set. In this embodiment, the obstacle grounding point data set contains the obstacle type and the obstacle to which the docking location belongs, and the initial obstacle grounding point detection model is trained by the marked obstacle grounding point data set to generate the above-mentioned obstacle grounding point detection model.

[0086] Static obstacles in the multi-camera image are detected by the obstacle detection model to determine multiple grounding points of the static obstacles. It should be noted that, in this embodiment, when the obstacle detection model generates multiple grounding points, the obstacles to which each grounding point belongs and the obstacle categories to which it belongs are marked to distinguish each grounding point. For example, there are static obstacles wall 1, wall 2, stone pillar 1, and bush 1 in the multi-camera image. When the static obstacles in the multi-camera image are detected by the obstacle detection model, multiple grounding points of the obstacles are determined. When the grounding points in the multi-camera image are determined, the static obstacle docking locations corresponding to the grounding points are marked, for example, as grounding point 1 (wall 1), grounding point 2 (wall 1), etc. It can be understood that when the obstacle detection model in this embodiment detects the grounding point of the static obstacle in the image, the grounding point is marked at the corresponding position in the multi-camera image to identify the position of the grounding point in the multi-camera image, and the obstacle to which the grounding point belongs and the category of the obstacle to which it belongs are marked by annotation.

[0087] In step S13, a second 3D detection frame of the static obstacle is determined according to the multiple grounding points.

[0088] For example, in this embodiment, according to the marked positions of each grounding point in the above steps and the marked information of each grounding point, the grounding points belonging to the same obstacle are classified into a type of grounding points, and a corresponding grounding point data set is generated. The grounding line of the corresponding obstacle is determined based on the grounding point data set, and then the second 3D detection frame of the static obstacle is generated in the multi-camera image according to the category of the obstacle. For example, in this embodiment, the vehicle terminal is loaded with 3D size information of different types of static obstacles, and the 3D size information includes information such as the length, width, and height of the static obstacle in the physical space. The 3D size information is converted based on the size ratio of the multi-camera image to generate the size parameters of the static obstacle in the multi-camera image. The 3D space occupied by the static obstacle in the multi-camera image is completed based on the grounding line and size parameters of the obstacle, thereby generating the 3D size information of the static obstacle.

[0089] In step S14, the first 3D detection frame and the second 3D detection frame are projected into the image coordinate system of the multi-camera image to generate multiple 2D detection frames in the image coordinate system.

[0090] For example, in this embodiment, after determining the first 3D detection frame of the dynamic obstacle in the multi-camera image and the second 3D detection frame of the static obstacle in the above method, based on the intrinsic parameters and extrinsic parameters of the corresponding camera when generating the detection frame, the 3D detection frame is projected and transformed, and the 3D detection frame is projected into the image coordinate system of the multi-camera image to generate multiple 2D detection frames in the image coordinate system.

[0091] In step S15, the obstacle occlusion rate of the dynamic obstacle is determined according to the overlap rate between the 2D detection frames in the plurality of 2D detection frames.

[0092] For example, in this embodiment, the target 2D detection frame corresponding to the dynamic obstacle is first determined from the 2D detection frame, and then the overlap rate between the target 2D detection frame and other 2D detection frames is determined, and the obstacle occlusion rate of the dynamic obstacle is determined according to the overlap rate.

[0093] In the above manner, in response to detecting a dynamic obstacle in a multi-camera image, the dynamic obstacle is detected by an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle, the static obstacle in the multi-camera image is detected by an obstacle ground point detection model to determine multiple ground points of the static obstacle, and a second 3D detection frame of the static obstacle is determined based on the multiple ground points, and the first 3D detection frame and the second 3D detection frame are projected into the image coordinate system of the multi-camera image to generate multiple 2D detection frames in the image coordinate system, and the obstacle occlusion rate of the dynamic obstacle is determined based on the overlap rate between each 2D detection frame in the multiple 2D detection frames. Thus, information of obstacles other than the dynamic obstacle to be detected is determined, the detection frames of each obstacle are determined by fitting, the obstacle occlusion rate of the dynamic obstacle is determined based on the overlap rate between the detection frames, and the calculation is performed based on the detection frame, so that the determined visual obstacle occlusion rate is more accurate.

[0094] Figure 2 is a flow chart of a method for determining an obstacle occlusion rate according to an exemplary embodiment. Figure 2 As shown, the above step S15 includes the following steps.

[0095] In step S151 , a target 2D detection frame of a dynamic obstacle is determined from multiple 2D detection frames.

[0096] For example, in this embodiment, a special mark is used to mark the first 3D detection frame of the dynamic obstacle, so as to determine the target 2D detection frame corresponding to the dynamic obstacle among the generated multiple 2D detection frames.

[0097] In step S152 , based on the target 2D detection frame, a first 2D detection frame that meets a preset condition is screened out from multiple 2D detection frames.

[0098] It should be noted that in the process of generating 3D detection frames, the 3D detection frames in the same multi-camera image can come from different camera devices. Therefore, in order to distinguish the 3D detection frames, the 3D detection frames are labeled based on the corresponding camera IDs in the process of generating 3D detection frames to distinguish the different sources of the 3D detection frames. In contrast, when generating 2D detection frames, the 2D detection frames are also labeled based on the camera ID information. For example, in this embodiment, based on the target 2D detection frame, the detection frames that will cause occlusion of the target 2D detection frame are screened from multiple 2D detection frames, thereby determining the first 2D detection frame that meets the preset conditions. Among them, the screening conditions include at least:

[0099] (1) The camera ID corresponding to the 2D detection frame is the same as the camera ID of the target 2D detection frame;

[0100] (2) The depth value of the camera corresponding to the 2D detection frame is less than or equal to the depth value of the camera corresponding to the target 2D detection frame;

[0101] (3) The diagonal coordinates of the 2D detection frame in the multi-camera image are different from the diagonal coordinates of the target 2D detection frame in the multi-camera image;

[0102] (4) The X value and Y value of the lower right corner coordinates of the 2D detection frame are less than or equal to the X value and Y value of the upper left corner coordinates of the target detection frame.

[0103] Through the above conditions, multiple 2D detection frames are screened based on the target 2D detection frame, and a first 2D detection frame that meets the preset condition is determined from the multiple 2D detection frames.

[0104] In step S153 , the obstacle occlusion rate is determined according to the overlap rate between the target 2D detection frame and the first 2D detection frame.

[0105] For example, the overlap ratio between the target 2D detection frame and the first 2D detection frame is calculated, and the obstacle occlusion ratio is determined based on the overlap ratio. The method of determining the obstacle occlusion ratio based on the overlap ratio is the same as that in the above step S15, and can refer to the above step S15, which will not be repeated here.

[0106] Optionally, in one implementation, the above step S153 includes:

[0107] Determine a plurality of first overlapping areas between the target 2D detection frame and the first 2D detection frame;

[0108] determining a plurality of second overlapping areas between the plurality of first overlapping areas;

[0109] Determining a target overlapping area according to the plurality of first overlapping areas and the plurality of second overlapping areas;

[0110] The obstacle occlusion rate is determined based on the target overlapping area and the initial area of ​​the target 2D detection box.

[0111] For example, in this embodiment, multiple first overlapping areas between the target 2D detection frame and other first 2D detection frames in the image coordinate system are determined, and the first overlapping areas are summed, wherein there is an overlapping portion between the first overlapping area and the first overlapping area, and thus multiple second overlapping areas between the multiple first overlapping areas are determined. After summing the multiple first overlapping areas, the sum of the overlapping areas is subtracted from the sum of the second overlapping areas, thereby determining the target overlapping area between the target 2D detection frame and the other 2D detection frames, and generating the obstacle occlusion rate of the dynamic obstacle based on the target overlapping area and the initial area of ​​the target 2D detection frame.

[0112] By using the above method, the generated multiple 2D detection frames are screened to determine the first 2D detection frame that meets the conditions, so that the obtained occlusion rate data is more accurate.

[0113] Figure 3 is a flow chart of a method for determining a 3D detection frame according to an exemplary embodiment. Figure 3 As shown, the above step S13 includes the following steps.

[0114] In step S131, cluster analysis is performed on the multiple grounding points to determine multiple grounding point sets, and each grounding point in each grounding point set belongs to the same static obstacle.

[0115] By way of example, in this embodiment, after multiple grounding points are determined through the above steps, cluster analysis is performed on the multiple grounding points according to the categories of the grounding points and the static obstacles to generate multiple grounding point sets of different static obstacle categories.

[0116] Optionally, in one implementation, the above step S131 includes:

[0117] Determine a plurality of grounding point categories corresponding to each of the plurality of grounding points;

[0118] determining a neighborhood radius and an object quantity threshold of a first grounding point among the multiple grounding points according to a first grounding point category of the first grounding point, the first grounding point being any one of the multiple grounding points;

[0119] If the number of other grounding points within the neighborhood radius of the first grounding point is greater than or equal to the object number threshold, the first grounding point is determined to be the core grounding point of the first grounding point set, and the grounding point categories of other grounding points are all the first grounding point category;

[0120] Multiple grounding point sets are generated according to multiple core grounding points.

[0121] For example, in this embodiment, the grounding point category of each grounding point is determined according to the above steps. For grounding points of different grounding point categories, the thresholds for the neighborhood radius and the number of detected objects within the neighborhood radius are different. Whether the currently detected grounding point is a core point is determined based on the number of detected objects in the neighborhood radius.

[0122] In this embodiment, after determining the neighborhood radius of the grounding point, the number of other grounding points of the same type within the neighborhood radius of the grounding point is determined based on any grounding point. When the number is greater than or equal to the object number threshold, the grounding point is determined to be the core grounding point of this type of static obstacle. In this way, other grounding points of the same static obstacle are determined, thereby generating a grounding point set of the static obstacle.

[0123] In step S132, a first grounding line of a first static obstacle corresponding to the first grounding point set is determined according to a first grounding point set among the multiple grounding point sets, where the first grounding point set is any grounding point set among the multiple grounding point sets.

[0124] For example, in this embodiment, for any first grounding point set in multiple grounding point sets, a clustering algorithm is used to determine a first grounding line of a first static obstacle corresponding to the first grounding point set. For example, in this embodiment, multiple grounding point sets can be connected in sequence to generate a first grounding line corresponding to the first grounding point set.

[0125] In step S133, the length information of the first grounding wire, the heading angle information, and the obstacle category information of the first static obstacle are determined.

[0126] In step S134, the preset three-dimensional size information of the first static obstacle is determined according to the obstacle category information.

[0127] For example, in this embodiment, the length information, heading angle information and obstacle category information of the first static obstacle corresponding to the first grounding line are determined based on the generated first grounding line. Based on this information, the preset three-dimensional size information matching the obstacle category is determined from the vehicle terminal. In this embodiment, the three-dimensional sizes of different types of static obstacles can be counted, and a mapping table can be loaded in the vehicle terminal. The mapping table includes a one-to-one correspondence between static obstacles and three-dimensional sizes. Based on the obstacle category information of the static obstacle, the preset three-dimensional size information of the first static obstacle is determined.

[0128] In step S135 , a 3D detection frame of the first static obstacle is generated according to the length information, the heading angle information and the preset three-dimensional size information.

[0129] For example, in this embodiment, the 3D detection frame of the static obstacle in the multi-camera image is completed according to the length information, the heading angle information and the preset three-dimensional size information, so as to generate a 3D detection frame of the first static obstacle.

[0130] Through the above method, the 3D detection frame of the static obstacle is determined, and the 3D detection frame of the static obstacle is marked in the multi-camera image based on the ground line of the static obstacle. The occlusion of the dynamic obstacle by the static obstacle is considered, so that the obstacle occlusion rate finally obtained is more accurate.

[0131] Figure 4 is a block diagram of a device for determining an obstacle occlusion rate according to an exemplary embodiment. Figure 4 The device 100 includes: a first generating module 110, a first determining module 120, a second determining module 130, a second generating module 140 and an executing module 150.

[0132] The first generating module 110 is configured to detect the dynamic obstacle by using an obstacle detection model in response to detecting the dynamic obstacle in the multi-camera image, so as to generate a first 3D detection frame of the dynamic obstacle;

[0133] The first determination module 120 is configured to detect static obstacles in the multi-camera image through an obstacle grounding point detection model to determine multiple grounding points of the static obstacles;

[0134] The second determination module 130 is configured to determine a second 3D detection frame of the static obstacle according to the multiple grounding points;

[0135] The second generating module 140 is configured to project the first 3D detection frame and the second 3D detection frame into the image coordinate system of the multi-camera image to generate a plurality of 2D detection frames in the image coordinate system;

[0136] The execution module 150 is configured to determine the obstacle occlusion rate of the dynamic obstacle according to the overlap rate between each 2D detection frame in the multiple 2D detection frames.

[0137] Optionally, the execution module 150 includes:

[0138] A first determination submodule is configured to determine a target 2D detection frame of a dynamic obstacle from multiple 2D detection frames;

[0139] A selection submodule is configured to select a first 2D detection frame that meets a preset condition from multiple 2D detection frames based on the target 2D detection frame;

[0140] The second determination submodule is configured to determine the obstacle occlusion rate according to the overlap rate between the target 2D detection frame and the first 2D detection frame.

[0141] Optionally, the second determining submodule is configured to:

[0142] Determine a plurality of first overlapping areas between the target 2D detection frame and the first 2D detection frame;

[0143] determining a plurality of second overlapping areas between the plurality of first overlapping areas;

[0144] Determining a target overlapping area according to the plurality of first overlapping areas and the plurality of second overlapping areas;

[0145] The obstacle occlusion rate is determined based on the target overlapping area and the initial area of ​​the target 2D detection box.

[0146] Optionally, the device 100 further includes a training module, the training module including:

[0147] An acquisition submodule, configured to acquire an initial obstacle detection data set and laser point cloud data at the same time;

[0148] The first generation submodule is configured to annotate the obstacle 3D detection frame of the initial obstacle detection data set in the initial multi-camera image according to the laser point cloud data to generate an initial obstacle detection training set;

[0149] The second generation submodule is configured to amplify the initial obstacle detection training set by using a preset image transformation rule to generate a target obstacle detection training set, wherein the preset image transformation rule includes at least one of image frame parameter transformation, image block exchange, image cropping transformation, and image block occlusion transformation;

[0150] The third generation submodule is configured to train an initial obstacle detection model based on the target obstacle detection training set to generate an obstacle detection model.

[0151] Optionally, the training module further includes a screening submodule, and the screening submodule is configured to:

[0152] Obtaining camera parameters of an initial multi-camera image, where the camera parameters include at least camera intrinsic parameters and camera extrinsic parameters;

[0153] According to the camera parameters, the obstacle 3D detection frame is projected into the image coordinate system of the initial multi-camera image to generate multiple 2D detection frames;

[0154] Determine a plurality of intersection-over-union ratios between each of the plurality of 2D detection frames;

[0155] When a first intersection-over-union ratio of a first 2D detection frame among multiple 2D detection frames is greater than a set threshold, a first 3D detection frame corresponding to the first 2D detection frame is deleted from the labeled initial multi-camera image, where the first 2D detection frame is any 2D detection frame among the multiple 2D detection frames.

[0156] Optionally, the second determining module 130 includes:

[0157] A third determination submodule is configured to perform cluster analysis on the plurality of grounding points to determine a plurality of grounding point sets, wherein each grounding point in each grounding point set belongs to the same static obstacle;

[0158] a fourth determination submodule, configured to determine, according to a first grounding point set among the plurality of grounding point sets, a first grounding line of the first static obstacle corresponding to the first grounding point set, wherein the first grounding point set is any grounding point set among the plurality of grounding point sets;

[0159] a fifth determination submodule, configured to determine length information of the first grounding wire, heading angle information, and obstacle category information of the first static obstacle;

[0160] a sixth determination submodule, configured to determine preset three-dimensional size information of the first static obstacle according to the obstacle category information;

[0161] The generating submodule is configured to generate a 3D detection frame of the first static obstacle according to the length information, the heading angle information and the preset three-dimensional size information.

[0162] Optionally, the third determining submodule is configured to:

[0163] Determine a plurality of grounding point categories corresponding to each of the plurality of grounding points;

[0164] determining a neighborhood radius and an object quantity threshold of a first grounding point among the multiple grounding points according to a first grounding point category of the first grounding point, the first grounding point being any one of the multiple grounding points;

[0165] If the number of other grounding points within the neighborhood radius of the first grounding point is greater than or equal to the object number threshold, the first grounding point is determined to be the core grounding point of the first grounding point set, and the grounding point categories of other grounding points are all the first grounding point category;

[0166] Multiple grounding point sets are generated according to multiple core grounding points.

[0167] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0168] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the steps of the method for determining the obstacle occlusion rate provided by the present disclosure.

[0169] Figure 5 is a block diagram of a vehicle 500 according to an exemplary embodiment. For example, the vehicle 500 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 500 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0170] Reference Figure 5 , the vehicle 500 may include various subsystems, for example, an infotainment system 510, a perception system 520, a decision control system 530, a drive system 540, and a computing platform 550. The vehicle 500 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 500 may be interconnected by wire or wireless means.

[0171] In some embodiments, the infotainment system 510 may include a communication system, an entertainment system, and a navigation system, among others.

[0172] The perception system 520 may include several sensors for sensing information about the environment around the vehicle 500. For example, the perception system 520 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0173] The decision control system 530 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0174] The drive system 540 may include components that provide powered motion for the vehicle 500. In one embodiment, the drive system 540 may include an engine, an energy source, a transmission system, and wheels. The engine may be an internal combustion engine, an electric motor, an air compression engine, or a combination of one or more thereof. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0175] Some or all functions of the vehicle 500 are controlled by a computing platform 550. The computing platform 550 may include at least one processor 551 and a memory 552, and the processor 551 may execute instructions 553 stored in the memory 552.

[0176] The processor 551 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0177] The memory 552 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0178] In addition to instructions 553 , memory 552 may also store data, such as road maps, route information, vehicle location, direction, speed, etc. The data stored in memory 552 may be used by computing platform 550 .

[0179] In the embodiment of the present disclosure, the processor 551 may execute instruction 553 to complete all or part of the steps of the above-mentioned method for determining the obstacle occlusion rate.

[0180] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-mentioned obstacle occlusion rate determination method when executed by the programmable device.

[0181] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0182] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for determining an obstacle occlusion rate, characterized in that: include: In response to detecting a dynamic obstacle in the multi-camera image, detecting the dynamic obstacle by using an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle; Detecting static obstacles in the multi-camera image using an obstacle grounding point detection model to determine multiple grounding points of the static obstacles; Determining a second 3D detection frame of the static obstacle according to the multiple grounding points; Projecting the first 3D detection frame and the second 3D detection frame into an image coordinate system of the multi-camera image to generate a plurality of 2D detection frames in the image coordinate system; An obstacle occlusion rate of the dynamic obstacle is determined according to an overlap rate between each 2D detection frame in the multiple 2D detection frames.

2. The method according to claim 1, characterized in that: The determining the obstacle occlusion rate of the dynamic obstacle according to the overlap rate between each 2D detection frame in the multiple 2D detection frames includes: Determine a target 2D detection frame of the dynamic obstacle from the multiple 2D detection frames; Based on the target 2D detection frame, filter out a first 2D detection frame that meets a preset condition from the multiple 2D detection frames; The obstacle occlusion rate is determined according to an overlap rate between the target 2D detection frame and the first 2D detection frame.

3. The method according to claim 2, characterized in that The determining the obstacle occlusion rate according to the overlap rate between the target 2D detection frame and the first 2D detection frame includes: Determine a plurality of first overlapping areas between the target 2D detection frame and the first 2D detection frame; determining a plurality of second overlapping areas between the plurality of first overlapping areas; Determining a target overlapping area according to the plurality of first overlapping areas and the plurality of second overlapping areas; The obstacle occlusion rate is determined according to the target overlapping area and the initial area of ​​the target 2D detection frame.

4. The method according to claim 1, characterized in that The obstacle detection model is trained and generated in the following way: Obtain the initial obstacle detection data set and the laser point cloud data at the same time; According to the laser point cloud data, annotating the obstacle 3D detection frame of the initial obstacle detection data set in the initial multi-camera image to generate an initial obstacle detection data set; The initial obstacle detection data set is augmented by a preset image transformation rule to generate a target obstacle detection data set, wherein the preset image transformation rule includes at least one of image frame parameter transformation, image block exchange, image cropping transformation, and image block occlusion transformation; An initial obstacle detection model is trained based on the target obstacle detection data set to generate the obstacle detection model.

5. The method according to claim 4, characterized in that The method further comprises: Acquire camera parameters of the initial multi-camera image, wherein the camera parameters include at least camera intrinsic parameters and camera extrinsic parameters; According to the camera parameters, the obstacle 3D detection frame is projected into the image coordinate system of the initial multi-camera image to generate multiple 2D detection frames; Determine a plurality of intersection-over-union ratios between each of the plurality of 2D detection frames; When a first intersection-over-union ratio of a first 2D detection frame among the multiple 2D detection frames is greater than a set threshold, a first 3D detection frame corresponding to the first 2D detection frame is deleted from the labeled initial multi-camera image, where the first 2D detection frame is any 2D detection frame among the multiple 2D detection frames.

6. The method according to claim 1, characterized in that The determining, according to the multiple grounding points, a second 3D detection frame of the static obstacle includes: Performing cluster analysis on the multiple grounding points to determine multiple grounding point sets, each grounding point in each grounding point set belonging to the same static obstacle; Determine, according to a first grounding point set among the multiple grounding point sets, a first grounding line of a first static obstacle corresponding to the first grounding point set, wherein the first grounding point set is any grounding point set among the multiple grounding point sets; Determining the length information and heading angle information of the first grounding wire and the obstacle category information of the first static obstacle; Determining preset three-dimensional size information of the first static obstacle according to the obstacle category information; A 3D detection frame of the first static obstacle is generated according to the length information, the heading angle information and the preset three-dimensional size information.

7. The method according to claim 6, characterized in that The performing cluster analysis on the plurality of grounding points to determine a plurality of grounding point sets includes: Determine a plurality of grounding point categories corresponding to each of the plurality of grounding points; determining a neighborhood radius and an object quantity threshold of a first grounding point among the multiple grounding points according to a first grounding point category of the first grounding point, wherein the first grounding point is any one of the multiple grounding points; If the number of other grounding points within the neighborhood radius of the first grounding point is greater than or equal to the object number threshold, the first grounding point is determined to be a core grounding point of the first grounding point set, and the grounding point categories of the other grounding points are all the first grounding point category; The plurality of grounding point sets are generated according to the plurality of core grounding points.

8. A device for determining an obstacle occlusion rate, characterized in that: The device comprises: A first generating module is configured to detect a dynamic obstacle in the multi-camera image by using an obstacle detection model to generate a first 3D detection frame of the dynamic obstacle; A first determination module is configured to detect static obstacles in the multi-camera image through an obstacle grounding point detection model to determine multiple grounding points of the static obstacles; A second determination module is configured to determine a second 3D detection frame of the static obstacle according to the multiple grounding points; a second generating module, configured to project the first 3D detection frame and the second 3D detection frame into an image coordinate system of the multi-camera image to generate a plurality of 2D detection frames in the image coordinate system; The execution module is configured to determine the obstacle occlusion rate of the dynamic obstacle according to the overlap rate between each 2D detection frame in the multiple 2D detection frames.

9. A vehicle, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the executable instructions to implement the steps of the method for determining the obstacle occlusion rate as described in any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method for determining the obstacle occlusion rate described in any one of claims 1 to 7 are implemented.

Images