Detection method and device applied to height limiting object of vehicle, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a detection method and device applied to a height limiting object of a vehicle, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring an image and initial point cloud data of the driving front of a vehicle, and determining a bounding box of a height limiting object in the image; and determining the point cloud data of the height limiting object from the initial point cloud data according to the bounding box. And according to the point cloud data of the height limiting object and the height information of the vehicle, generating a detection result representing whether the vehicle can pass through the height limiting object. The method is used for achieving the effect of accurately detecting whether the vehicle can pass through the height limiting object or not.

Description

Technical Field

[0001] The present application relates to the field of vehicle autonomous driving, and in particular to a detection method, device, electronic device, storage medium and program product for height-limited objects applied to vehicles. Background Art

[0002] During the driving process of the vehicle, the vehicle will pass through some height-restricted target objects (ie, height-restricted objects), for example, the height-restricted target objects are height-restricted poles, bridges, tunnels, underground parking lots, etc. It is necessary to detect whether the vehicle can pass through the height-restricted objects.

[0003] Furthermore, there is an urgent need for a solution that can accurately detect whether a vehicle can pass through a height-restricted object. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, storage medium and program product for detecting a height-restricted object applied to a vehicle, so as to achieve the effect of accurately detecting whether a vehicle can pass through the height-restricted object.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting a height-restricted object applied to a vehicle, comprising:

[0006] An image and initial point cloud data of the front of the vehicle are obtained, and a bounding box of a height-restricted object in the image is determined.

[0007] According to the bounding box, point cloud data of the height-limited object is determined from the initial point cloud data.

[0008] Based on the point cloud data of the height-restricted object and the height information of the vehicle, a detection result is generated to indicate whether the vehicle can pass through the height-restricted object.

[0009] In a possible implementation, determining point cloud data of the height-limited object from initial point cloud data according to the bounding box includes:

[0010] The two-dimensional coordinates of the bounding box are converted to the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box.

[0011] Based on the point cloud coordinates of the bounding box, the point cloud data outside the bounding box is filtered out from the initial point cloud data to obtain the point cloud data of the height-limited object.

[0012] In a possible implementation, converting the two-dimensional coordinates of the bounding box into a three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box includes:

[0013] The two-dimensional coordinates of the bounding box are calculated and processed according to the preset external parameter matrix and the preset internal parameter matrix to obtain the point cloud coordinates of the bounding box.

[0014] In a possible implementation, the two-dimensional coordinates of the bounding box include the two-dimensional coordinates of two corner points on a diagonal line of the bounding box.

[0015] In a possible implementation, generating a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle includes:

[0016] The point cloud data of the height-restricted object is clustered to obtain a three-dimensional contour point cloud set of the height-restricted object.

[0017] Generate detection results based on the 3D contour point cloud set and the vehicle's height information.

[0018] In a possible implementation, generating a detection result according to the three-dimensional contour point cloud set and the height information of the vehicle includes:

[0019] The 3D contour point cloud with the smallest height in the 3D contour point cloud set is determined as the first point cloud of the height-limited object.

[0020] From the first point clouds of the height-limited objects, the first point cloud closest to the vehicle is determined as the second point cloud.

[0021] A detection result is generated based on the second point cloud and the height information of the vehicle.

[0022] In a possible implementation, generating a detection result according to the second point cloud and the height information of the vehicle includes:

[0023] If it is determined that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle can pass through the height-restricted object.

[0024] If it is determined that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle cannot pass through the height-restricted object.

[0025] In a possible implementation, obtaining initial point cloud data ahead of the vehicle includes:

[0026] Initial point cloud data collected by the 4D millimeter-wave radar on the vehicle.

[0027] In a possible implementation, determining a bounding box of a height-limited object in an image includes:

[0028] The image is input into the preset deep learning model for feature extraction to obtain image feature data.

[0029] The image feature data is recognized and processed based on the preset deep learning model to obtain the bounding box of the height-restricted object.

[0030] In a possible implementation, the height-restricted object has a category; and the method further includes:

[0031] Generate and issue a prompt message, wherein the prompt message includes the category and detection result of the height-restricted object.

[0032] In a second aspect, an embodiment of the present application provides a detection device for a height-limited object applied to a vehicle, comprising:

[0033] The acquisition unit is used to acquire the image and initial point cloud data of the front of the vehicle.

[0034] The first determining unit is used to determine a bounding box of a height-limited object in an image.

[0035] The second determining unit is used to determine the point cloud data of the height-limited object from the initial point cloud data according to the bounding box.

[0036] The generating unit is used to generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

[0037] In a possible implementation manner, the second determining unit includes:

[0038] The conversion module is used to convert the two-dimensional coordinates of the bounding box into the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box.

[0039] The filtering module is used to filter out the point cloud data outside the bounding box from the initial point cloud data based on the point cloud coordinates of the bounding box to obtain the point cloud data of the height-limited object.

[0040] In a possible implementation manner, the conversion module is specifically configured to:

[0041] The two-dimensional coordinates of the bounding box are calculated and processed according to the preset external parameter matrix and the preset internal parameter matrix to obtain the point cloud coordinates of the bounding box.

[0042] In a possible implementation, the two-dimensional coordinates of the bounding box include the two-dimensional coordinates of two corner points on a diagonal line of the bounding box.

[0043] In a possible implementation, the generating unit includes:

[0044] The clustering module is used to perform clustering processing on the point cloud data of the height-limited object to obtain a three-dimensional contour point cloud set of the height-limited object.

[0045] The generation module is used to generate detection results based on the three-dimensional contour point cloud set and the height information of the vehicle.

[0046] In a possible implementation, the generating module includes:

[0047] The first determination submodule is used to determine the three-dimensional contour point cloud with the smallest height in the three-dimensional contour point cloud set as the first point cloud of the height-limited object.

[0048] The second determination submodule is used to determine, from the first point clouds of the height-limited objects, a first point cloud that is closest to the vehicle as the second point cloud.

[0049] The generation submodule is used to generate a detection result according to the second point cloud and the height information of the vehicle.

[0050] In a possible implementation, a submodule is generated, specifically for:

[0051] If it is determined that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle can pass through the height-restricted object.

[0052] If it is determined that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle cannot pass through the height-restricted object.

[0053] In a possible implementation, the acquiring unit includes:

[0054] The acquisition module is used to collect the initial point cloud data through the 4D millimeter wave radar on the vehicle.

[0055] In a possible implementation, the first determining unit includes:

[0056] The extraction module is used to input the image into a preset deep learning model for feature extraction to obtain image feature data.

[0057] The processing module is used to identify and process the image feature data based on a preset deep learning model to obtain a bounding box of the height-restricted object.

[0058] In a possible implementation manner, the height-restricted object has a category; and the device further includes:

[0059] The prompt unit is used to generate and issue prompt information, wherein the prompt information includes the category and detection result of the height-restricted object.

[0060] In a third aspect, an embodiment of the present application provides a detection device for height-restricted objects applied to a vehicle, including: a memory and a processor.

[0061] The memory stores computer-executable instructions.

[0062] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0064] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0065] The embodiments of the present application provide a method, device, electronic device, storage medium and program product for detecting height-restricted objects applied to vehicles, which collect two-dimensional images through a monocular camera and three-dimensional initial point cloud data through a vehicle-mounted radar, and obtain environmental information in front of the vehicle from different dimensions. Compared with the use of a single binocular camera or a lidar, it is less affected by lighting conditions or bad weather, thereby improving the accuracy and robustness of height-restricted object detection.

[0066] In addition, the embodiment of the present application processes the two-dimensional image captured by the monocular camera to obtain the bounding box of the height-restricted object in the image. Based on the bounding box of the height-restricted object and the initial point cloud data, the point cloud data corresponding to the height-restricted object in the initial point cloud data can be determined; thus, the point cloud data corresponding to the height-restricted object is compared with the height information of the vehicle to obtain the detection result of whether the vehicle can safely pass through the height-restricted object. This can avoid the interference of point cloud data that does not belong to the height-restricted object, and only judge the point cloud data corresponding to the height-restricted object, which can not only save the computing power resources of the on-board computer, but also reduce the interference of irrelevant data, and improve the pertinence of data processing and the accuracy of height-restricted object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of a method for detecting height-restricted objects applied to a vehicle provided in this application Figure 1 ;

[0069] Figure 2 A schematic diagram of a method for detecting height-restricted objects applied to a vehicle provided in this application Figure 2 ;

[0070] Figure 3 A schematic diagram of a detection device for a height-limited object applied to a vehicle provided in the present application Figure 1 ;

[0071] Figure 4 A schematic diagram of a detection device for a height-limited object applied to a vehicle provided in the present application Figure 2 ;

[0072] Figure 5 A schematic structural diagram of a detection device for height-limited objects applied to vehicles provided in the present application.

[0073] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0075] During the driving process, the vehicle will encounter various height-restricted objects, such as height-restricted poles, bridges, tunnels, and underground parking lots. These height-restricted objects put forward specific requirements for whether the vehicle can pass safely. Therefore, it is necessary to detect the height-restricted objects encountered during the driving process of the vehicle, and use the detection results of the height-restricted objects as the basis for judging whether the vehicle can safely pass through the height-restricted objects.

[0076] In one example, a binocular camera is deployed in a vehicle to detect the passable area of ​​the road in front of the vehicle. By comparing the height of the vehicle with the height and width of the passable area, it is determined whether the vehicle can pass normally.

[0077] However, in the above scheme, the binocular camera is greatly affected by the ambient light. When the light is too strong or too weak, the quality of the image collected by the binocular camera will decrease, which will in turn affect the determination of the height and width of the passable area, thereby reducing the accuracy of identifying the passable area.

[0078] In one example, a binocular camera and a laser radar are deployed in a vehicle to detect the height of a height limit pole in front of the vehicle. By comparing the height of the vehicle with the height of the height limit pole, it is determined whether the vehicle can pass smoothly.

[0079] However, in the above scheme, in adverse weather conditions such as rain, fog, and snow, the laser emitted by the lidar will be scattered by water droplets in the atmosphere, thereby affecting the propagation of the laser and reducing the detection accuracy of height-restricted objects.

[0080] The present application provides a method for detecting height-restricted objects applied to vehicles, aiming to solve the above technical problems in the prior art.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 1 The present application provides a method for detecting height-restricted objects applied to a vehicle, such as Figure 1 As shown, the method includes:

[0083] S101, obtaining an image and initial point cloud data in front of the vehicle, and determining a bounding box of a height-restricted object in the image.

[0084] For example, the execution subject of the embodiment of the present application may be an electronic device, or a server, or a terminal device, or other devices or equipment that can execute the solution of the embodiment, and this is not limited. This embodiment is introduced with the execution subject being an onboard computer.

[0085] A monocular camera is deployed in the vehicle, and the monocular camera continuously captures the environment in front of the vehicle to obtain an image in front of the vehicle, wherein the image in front of the vehicle includes one or more pictures. A monocular camera is an imaging device equipped with only one camera, and the environmental information in the image obtained is two-dimensional; in this embodiment, there is no limit on the number of monocular cameras deployed in the vehicle.

[0086] The vehicle is also equipped with an automotive radar, which transmits microwave signals (i.e., transmission signals) through antennas. The transmission signals are reflected by various targets in the environment to form echo signals. Based on the difference between the transmission signal and the echo signal, the three-dimensional coordinate information of each target in the environment can be calculated. The collection of three-dimensional coordinate information of each target in the environment (expressed in the form of target points) constitutes the initial point cloud data (PointCloudData). In other words, the initial point cloud data includes multiple target points, and each target point corresponds to its three-dimensional coordinate information.

[0087] After the on-board computer obtains the image sent by the monocular camera and the initial point cloud data sent by the on-board radar, it processes the image sent by the monocular camera to obtain the bounding box of the height-restricted object. The image can be processed by an existing deep learning model.

[0088] The bounding box of the height-limited object is a rectangular box surrounding the outside of the height-limited object in the image. The position of the height-limited object in the image can be determined through the coordinates of each corner point of the bounding box.

[0089] It is worth noting that there is a one-to-one correspondence between the height-restricted objects and the bounding boxes in the image, that is, each height-restricted object corresponds to a bounding box. If a picture includes multiple height-restricted objects, a corresponding bounding box will be determined for each height-restricted object.

[0090] S102 . Determine point cloud data of the height-limited object from the initial point cloud data according to the bounding box.

[0091] Exemplarily, after determining the bounding box of each height-restricted object, the onboard computer can obtain the coordinates of each corner point of the bounding box of each height-restricted object based on the bounding box of each height-restricted object. Next, based on the coordinates of each corner point of the bounding box of each height-restricted object, the three-dimensional coordinate information of each target point in the initial point cloud is compared to determine the point cloud data corresponding to each height-restricted object in the initial point cloud data, as well as the point cloud data that does not belong to the height-restricted object (i.e., other point cloud data).

[0092] Finally, the point cloud data that does not belong to the height-restricted objects can be filtered out to save the computing power resources of the on-board computer and improve processing efficiency.

[0093] S103 . Generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

[0094] Exemplarily, the vehicle computer obtains the height information of the vehicle. For example, the height information of the vehicle may be pre-stored or manually input into the vehicle computer by the user; in addition, when the vehicle is a commercial vehicle (e.g., a truck), the height information of the vehicle is related to the cargo load of the commercial vehicle, and therefore, a height sensor may be arranged in the vehicle to continuously monitor the height of the vehicle, thereby obtaining the current height information of the vehicle. In this embodiment, there is no restriction on the method of obtaining the height information of the vehicle.

[0095] The on-board computer compares the point cloud data of the determined height-restricted object, for example, the height value in the three-dimensional coordinate information of the target point in the point cloud data, with the height information of the vehicle, and generates a detection result based on the comparison result; wherein the detection result includes whether the vehicle can pass through the height-restricted object.

[0096] The embodiment of the present application provides a method for detecting height-restricted objects applied to vehicles. A two-dimensional image is collected by a monocular camera, and three-dimensional initial point cloud data is collected by a vehicle-mounted radar, so as to obtain environmental information in front of the vehicle from different dimensions. Compared with the use of a single binocular camera or a lidar, the method is less affected by lighting conditions or bad weather, thereby improving the accuracy and robustness of height-restricted object detection.

[0097] In addition, the embodiment of the present application processes the two-dimensional image captured by the monocular camera to obtain the bounding box of the height-restricted object in the image. Based on the bounding box of the height-restricted object and the initial point cloud data, the point cloud data corresponding to the height-restricted object in the initial point cloud data can be determined; thus, the point cloud data corresponding to the height-restricted object is compared with the height information of the vehicle to obtain the detection result of whether the vehicle can safely pass through the height-restricted object. This can avoid the interference of point cloud data that does not belong to the height-restricted object, and only judge the point cloud data corresponding to the height-restricted object, which can not only save the computing power resources of the on-board computer, but also reduce the interference of irrelevant data, and improve the pertinence of data processing and the accuracy of height-restricted object detection.

[0098] Figure 2 The present application provides a method for detecting height-restricted objects applied to a vehicle, such as Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, a detection method for a height-restricted object applied to a vehicle is described in detail, and the method includes:

[0099] S201, obtaining an image and initial point cloud data in front of the vehicle, and determining a bounding box of a height-restricted object in the image.

[0100] For example, the execution subject of the embodiment of the present application may be an electronic device, or a server, or a terminal device, or other devices or equipment that can execute the solution of the embodiment, and this is not limited. This embodiment is introduced with the execution subject being an onboard computer.

[0101] For the specific content of this step, please refer to the relevant content of step S101 in Example 1, which will not be repeated here.

[0102] In one example, the initial point cloud data in front of the vehicle is obtained in step S201, and the initial point cloud data may be collected by a 4D millimeter wave radar on the vehicle.

[0103] Exemplarily, a 4D millimeter-wave radar is deployed in the vehicle. The 4D millimeter-wave radar scans the environment in front of the vehicle, thereby obtaining a set of three-dimensional coordinate information of each target (expressed in the form of target points) in the environment in front of the vehicle, thereby forming initial point cloud data.

[0104] The principle of 4D millimeter wave radar is that it can calculate the distance, azimuth, speed and altitude information of each target point by using the frequency modulated continuous wave mode of the Doppler principle. Specifically, the 4D millimeter wave radar includes a signal transceiver component and a signal processing component. The signal transceiver component includes one or more signal transceiver units. Each signal transceiver unit includes a millimeter wave integrated circuit (Monolithic Microwave Integrated Circuit, MMIC) and several baseband antennas (Baseband Antenna) derived from the millimeter wave integrated circuit. The baseband antenna includes two types of antennas, one is a transmitting antenna and the other is a receiving antenna. Among them, the number of signal transceiver units can be four.

[0105] After each transmitting antenna transmits a millimeter wave (i.e., a transmitting wave), it is reflected by an object in the environment to obtain multiple reflected waves, and each reflected wave is received by a receiving antenna. The signal transceiver component sends each transmitting wave and reflected wave to the signal processing component, and the signal processing component determines the distance, azimuth, speed and altitude information of each target point in the environment by analyzing the difference between the transmitting wave and the reflected wave, such as time difference, phase difference, and frequency difference. In addition, the signal processing component processes the distance, azimuth, speed and altitude information of each target point in the environment and converts it into initial point cloud data.

[0106] In one example, the step S201 of “determining a bounding box of a height-limited object in an image” may include the following process:

[0107] The image is input into the preset deep learning model for feature extraction to obtain image feature data.

[0108] The image feature data is recognized and processed based on the preset deep learning model to obtain the bounding box of the height-restricted object.

[0109] Exemplarily, the onboard computer inputs the image into a preset deep learning model, which includes an input layer, a hidden layer, and an output layer connected in sequence. The preset deep learning model can be an algorithm model of the YOLO (You Only Look Once, referred to as YOLO) series.

[0110] The input layer is used to receive the picture in the image and convert the picture into image data; wherein the image data can be an array.

[0111] The hidden layer is used to receive the image data output by the input layer, process the image data, and thus extract the features of the image data to obtain image feature data; wherein the image feature data represents the content of the picture.

[0112] The output layer processes the features of the image data to output the processing results, which include the bounding box of the height-restricted object and the category of the height-restricted object. For example, the category of the height-restricted object can be a height-restricted pole, a bridge hole, or a temporary facility.

[0113] S202 . Determine point cloud data of the height-limited object from the initial point cloud data according to the bounding box.

[0114] In one example, the two-dimensional coordinates of the bounding box include two-dimensional coordinates of two corner points on a diagonal line of the bounding box.

[0115] For example, the onboard computer can obtain the coordinates of the corner points of each bounding box in the image based on the bounding box of each height-restricted object. Each bounding box includes four corner points, and each corner point corresponds to a coordinate.

[0116] For example, the onboard computer can only obtain the coordinates of the two corner points on the diagonal of each bounding box. For example, the coordinates of the lower left corner point of the bounding box can be (u, v) and the coordinates of the upper right corner point can be (u+w, v+h); where w represents the width of the bounding box and h represents the height of the bounding box. Compared with the onboard computer obtaining the coordinates of the four corner points of each bounding box, obtaining the coordinates of the two corner points on the diagonal of the bounding box can reduce the number of corner points that the onboard computer needs to process, reduce the computational complexity, and improve the data processing speed.

[0117] Of course, the coordinates of the two corner points on the diagonal line of the bounding box may also be the coordinates of the upper left corner point and the lower right corner point of the bounding box, and this embodiment does not limit this.

[0118] Furthermore, based on the coordinates of the two corner points on the diagonal of each bounding box and compared with the three-dimensional coordinate information of each target point in the initial point cloud, each target point within the coordinate range of each bounding box can be determined, thereby determining the point cloud data corresponding to each height-limited object from the initial point cloud data.

[0119] In one example, step S202 may include the following process:

[0120] The two-dimensional coordinates of the bounding box are converted to the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box.

[0121] Based on the point cloud coordinates of the bounding box, the point cloud data outside the bounding box is filtered out from the initial point cloud data to obtain the point cloud data of the height-limited object.

[0122] In one example, in step S202, “converting the two-dimensional coordinates of the bounding box into the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box” may include the following process:

[0123] The two-dimensional coordinates of the bounding box are calculated and processed according to the preset external parameter matrix and the preset internal parameter matrix to obtain the point cloud coordinates of the bounding box.

[0124] Exemplarily, in the process of comparing the coordinates of two corner points on the diagonal line of each bounding box with the three-dimensional coordinate information of each target point in the initial point cloud data, the conversion of the coordinate system is also involved.

[0125] This is because the two-dimensional coordinates of the bounding box are based on the coordinate system of the monocular camera, while the three-dimensional coordinate information of each target point in the initial point cloud data is based on the coordinate system of the 4D millimeter-wave radar, and there is a difference between the coordinate system of the monocular camera and the coordinate system of the 4D millimeter-wave radar; after the coordinate system conversion, the two-dimensional coordinates of the bounding box can be compared and calculated with the three-dimensional coordinate information of the target point.

[0126] According to the properties of the monocular camera, such as the focal length and optical center of the monocular camera, the preset internal parameter matrix is ​​calibrated and recorded as K. According to the position relationship between the monocular camera and the 4D millimeter wave radar, the preset external parameter matrix is ​​determined through operations such as rotation and translation, which is recorded as T.

[0127] The above-mentioned preset internal parameter matrix and preset external parameter matrix are determined by the R&D personnel and stored in the vehicle in advance. The on-board computer retrieves the pre-stored preset internal parameter matrix and preset external parameter matrix to calculate the two-dimensional coordinates of the bounding box, thereby converting the two-dimensional coordinates of the bounding box to the coordinates in the coordinate system of the 4D millimeter wave radar, that is, converting to obtain the point cloud coordinates of the bounding box.

[0128] For example, as shown in the following formula, for each two-dimensional coordinate of the bounding box, the preset external parameter matrix and the preset internal parameter matrix are sequentially multiplied on the left to achieve the conversion of the coordinate system.

[0129]

[0130] In the above formula, x represents the horizontal coordinate in the coordinate system of the 4D millimeter wave radar, y represents the vertical coordinate in the coordinate system of the 4D millimeter wave radar, u represents the horizontal coordinate in the coordinate system of the monocular camera, and v represents the horizontal coordinate in the coordinate system of the monocular camera.

[0131] After obtaining the point cloud coordinates of each bounding box, the on-board computer obtains the point cloud coordinates of the corner points of the diagonal of each bounding box. Taking one of the bounding boxes as an example, the point cloud coordinates of the corner points of its diagonal are recorded as (x1, y1) and (x2, y2), respectively. According to the coordinate range determined by (x1, y1) and (x2, y2), the target point whose coordinates in the initial point cloud data belong to this range can be determined, as shown in the following formula.

[0132]

[0133] The target points whose coordinates in the initial point cloud data belong to the above range are the point cloud data corresponding to each height-limited object.

[0134] Next, the on-board computer filters out the point cloud data that does not correspond to the height-restricted object from the initial point cloud data based on the point cloud data corresponding to each height-restricted object, so that only the point cloud data of each height-restricted object remains in the point cloud data.

[0135] S203: Generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

[0136] In one example, clustering is performed on point cloud data of a height-restricted object to obtain a three-dimensional contour point cloud set of the height-restricted object.

[0137] Generate detection results based on the 3D contour point cloud set and the vehicle's height information.

[0138] Exemplarily, the onboard computer performs clustering processing on the point cloud data of each height-limited object obtained after filtering, thereby obtaining a three-dimensional contour point cloud set; wherein the three-dimensional contour point cloud set includes the three-dimensional coordinate information of each target point corresponding to each height-limited object, that is, the three-dimensional contour point cloud. The three-dimensional coordinate information can be expressed in the form of (x, y, z), wherein x represents the position coordinate of the target point in the longitudinal direction, y represents the position coordinate of the target point in the lateral direction, and z represents the position coordinate of the target point in the height direction.

[0139] For example, clustering processing of point cloud data of each height-restricted object may be performed by using a clustering algorithm. For example, the clustering algorithm may be a density-based spatial clustering algorithm with noise (Density-Based Spatial Clustering of Applications with Noise, DBSCAN).

[0140] Next, the on-board computer compares the point cloud data of each height-restricted object with the height information of the vehicle, and based on the comparison results, generates a detection result that indicates whether the vehicle can safely pass through the height-restricted object.

[0141] In one example, in step S203, “generating a detection result according to the three-dimensional contour point cloud set and the height information of the vehicle” may include the following process:

[0142] Determine the 3D contour point cloud with the smallest height in the 3D contour point cloud set as the first point cloud of the height-limited object.

[0143] From the first point clouds of the height-limited objects, the first point cloud closest to the vehicle is determined as the second point cloud.

[0144] A detection result is generated based on the second point cloud and the height information of the vehicle.

[0145] For example, there are often more than one height-restricted objects in front of the vehicle, and whether the vehicle can pass through the height-restricted object that is about to be encountered needs to be determined first. Therefore, it is necessary to determine the height-restricted object closest to the front of the vehicle and determine the height of the nearest height-restricted object.

[0146] First, the on-board computer determines the three-dimensional contour point cloud with the smallest height value from the three-dimensional contour point cloud set of each height-limited object, and records it as the first point cloud.

[0147] For example, the onboard computer traverses the coordinate values ​​of the height direction of each 3D contour point cloud in the 3D contour point cloud set of each height-limited object. For example, the coordinate value of the height direction may be the value of the z-coordinate, and the 3D contour point cloud with the smallest z-axis coordinate value is determined, which is recorded as the first point cloud. It is worth noting that during the driving process of the vehicle, there is often more than one height-limited object in front. Therefore, when there are multiple height-limited objects, a first point cloud will be determined for each 3D contour point cloud set of the height-limited object.

[0148] Next, after determining each first point cloud, the onboard computer determines the coordinates of each first point cloud in the x-axis direction, thereby determining the first point cloud with the closest distance to the front of the vehicle, which is recorded as the second point cloud. For example, the coordinate value in the x-axis direction of each first point cloud is subtracted from the coordinate value in the x-axis direction of the front of the vehicle to obtain the distance between each first point cloud and the front of the vehicle, and the first point cloud with the smallest distance is used as the second point cloud. In this embodiment, the direction of the x-axis is defined as the direction indicated by the vehicle's travel, that is, the direction of the front of the vehicle.

[0149] Subsequently, the on-board computer compares the coordinates of the height direction of the second point cloud with the height information of the vehicle. For example, the z coordinate value of the second point cloud is compared with the z coordinate value of the vehicle, and determines whether the vehicle can safely pass through the height-restricted object based on the relationship between the two values.

[0150] In one example, in step S203, “generating a detection result according to the second point cloud and the height information of the vehicle” may include the following process:

[0151] If it is determined that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle can pass through the height-restricted object.

[0152] If it is determined that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle cannot pass through the height-restricted object.

[0153] Exemplarily, the onboard computer compares the determined z-axis coordinate value of the second point cloud (ie, representing the height of the second point cloud) with the z-axis coordinate value in the height information of the vehicle.

[0154] If the coordinate system of the second point cloud is different from the coordinate system of the vehicle, for example, the coordinate system of the second point cloud is the coordinate system of the 4D millimeter-wave radar, and the coordinate system of the vehicle is the automobile coordinate system; then, the z-axis coordinate value of the second point cloud cannot be directly compared with the z-axis coordinate value in the height information of the vehicle; before comparison, it is necessary to unify the coordinates of the second point cloud and the height information of the vehicle, for example, through translation, rotation and other calculation operations, after making the coordinate systems of the two consistent, the z-axis coordinate value of the second point cloud cannot be directly compared with the z-axis coordinate value in the height information of the vehicle.

[0155] Through the above comparison, if the on-board computer determines that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the vehicle's height information, it indicates that the height-restricted object closest to the front of the vehicle is higher than the vehicle, and therefore, the vehicle can pass through the height-restricted object; the detection results output by the on-board computer include content indicating that the vehicle can pass through the height-restricted object.

[0156] If the on-board computer determines that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the vehicle's height information, it indicates that the height-restricted object closest to the front of the vehicle is lower than the height of the vehicle, and therefore, the vehicle cannot pass through the height-restricted object; the detection results output by the on-board computer include content indicating that the vehicle cannot pass through the height-restricted object.

[0157] S204: Generate and issue prompt information, wherein the prompt information includes the category and detection result of the height-restricted object. The height-restricted object has a category.

[0158] Exemplarily, the on-board computer generates and sends a prompt message to the user based on the content in the detection results and the category of the height-restricted object output by the preset deep learning model.

[0159] For example, the prompt information may be "The height restriction object ahead is a height restriction pole, and the vehicle can pass safely", or "The height restriction object ahead is a bridge tunnel, and the vehicle cannot pass safely"; the prompt information may be displayed in the form of text or graphics on the dashboard or central control screen of the vehicle, and the user obtains the prompt information by visual viewing, or the prompt information may also be sent to the vehicle's audio system through the onboard computer, and the prompt information is broadcasted through the vehicle's audio system, and the user obtains the prompt information by hearing, thereby knowing whether the vehicle can pass the height restriction object. In this embodiment, there is no restriction on the presentation form of the prompt information and the prompt method for the user.

[0160] The embodiment of the present application provides a method for detecting height-restricted objects applied to vehicles. A two-dimensional image is collected by a monocular camera, and three-dimensional initial point cloud data is collected by a 4D millimeter-wave radar, so as to obtain environmental information in front of the vehicle from different dimensions. Compared with the use of a single binocular camera or a lidar, the method is less affected by lighting conditions or bad weather, thereby improving the accuracy and robustness of height-restricted object detection. Moreover, compared with a lidar, a 4D millimeter-wave radar has a lower cost, which can improve the economy of the vehicle.

[0161] In addition, the embodiment of the present application uses a preset deep learning model to process the two-dimensional image collected by the monocular camera to obtain the bounding box of the height-limited object in the image. According to the bounding box of the height-limited object and the initial point cloud data, the point cloud data corresponding to the height-limited object in the initial point cloud data can be determined; thus, the point cloud data corresponding to the height-limited object is compared with the height information of the vehicle to obtain a detection result of whether the vehicle can safely pass through the height-limited object. On the one hand, this can avoid the interference of point cloud data that does not belong to the height-limited object, and only judge the point cloud data corresponding to the height-limited object, which can not only save the computing power resources of the on-board computer, but also reduce the interference of irrelevant data, improve the pertinence of data processing and the accuracy of detection of height-limited objects. On the other hand, because the preset deep learning model is obtained based on a large number of historical data training, it has high accuracy in identifying height-limited objects, thereby ensuring the accuracy of subsequent height judgments.

[0162] Figure 3 A schematic diagram of a detection device for a height-limited object applied to a vehicle provided in the present application Figure 1 , the device 30 comprises:

[0163] The acquisition unit 301 is used to acquire the image and initial point cloud data in front of the vehicle.

[0164] The first determining unit 302 is configured to determine a bounding box of a height-limited object in an image.

[0165] The second determining unit 303 is used to determine the point cloud data of the height-limited object from the initial point cloud data according to the bounding box.

[0166] The generating unit 304 is used to generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

[0167] The present embodiment provides a device for detecting height-restricted objects applied to vehicles, which can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in the present embodiment.

[0168] Figure 4 A schematic diagram of a detection device for a height-limited object applied to a vehicle provided in the present application Figure 2 , the device 40 comprises:

[0169] The acquisition unit 401 is used to acquire the image and initial point cloud data in front of the vehicle.

[0170] The first determining unit 402 is configured to determine a bounding box of a height-limited object in an image.

[0171] The second determining unit 403 is used to determine the point cloud data of the height-limited object from the initial point cloud data according to the bounding box.

[0172] The generating unit 404 is used to generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

[0173] In a possible implementation manner, the second determining unit 403 includes:

[0174] The conversion module 4031 is used to convert the two-dimensional coordinates of the bounding box into a three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box.

[0175] The filtering module 4032 is used to filter out the point cloud data outside the bounding box from the initial point cloud data based on the point cloud coordinates of the bounding box to obtain the point cloud data of the height-limited object.

[0176] In a possible implementation, the conversion module 4031 is specifically configured to:

[0177] The two-dimensional coordinates of the bounding box are calculated and processed according to the preset external parameter matrix and the preset internal parameter matrix to obtain the point cloud coordinates of the bounding box.

[0178] In a possible implementation, the two-dimensional coordinates of the bounding box include the two-dimensional coordinates of two corner points on a diagonal line of the bounding box.

[0179] In a possible implementation, the generating unit 404 includes:

[0180] The clustering module 4041 is used to perform clustering processing on the point cloud data of the height-limited object to obtain a three-dimensional contour point cloud set of the height-limited object.

[0181] The generation module 4042 is used to generate a detection result according to the three-dimensional contour point cloud set and the height information of the vehicle.

[0182] In a possible implementation, the generating module 4042 includes:

[0183] The first determination submodule 40421 is used to determine the three-dimensional contour point cloud with the smallest height in the three-dimensional contour point cloud set as the first point cloud of the height-limited object.

[0184] The second determination submodule 40422 is used to determine, from the first point clouds of the height-limited objects, the first point cloud that is closest to the vehicle as the second point cloud.

[0185] The generation submodule 40423 is used to generate a detection result according to the second point cloud and the height information of the vehicle.

[0186] In a possible implementation, the generating submodule 40423 is specifically used for:

[0187] If it is determined that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle can pass through the height-restricted object.

[0188] If it is determined that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle cannot pass through the height-restricted object.

[0189] In a possible implementation, the acquisition unit 401 includes:

[0190] The acquisition module 4011 is used to acquire the initial point cloud data through the 4D millimeter wave radar on the vehicle.

[0191] In a possible implementation manner, the first determining unit 402 includes:

[0192] The extraction module 4021 is used to input the image into a preset deep learning model for feature extraction to obtain image feature data.

[0193] The processing module 4022 is used to identify and process the image feature data based on a preset deep learning model to obtain a bounding box of the height-restricted object.

[0194] In a possible implementation manner, the height-restricted object has a category; and the device further includes:

[0195] The prompt unit 405 is used to generate and issue prompt information, wherein the prompt information includes the category and detection result of the height-restricted object.

[0196] The present embodiment provides a device for detecting height-restricted objects applied to vehicles, which can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in the present embodiment.

[0197] Figure 5 This is a schematic diagram of the structure of a detection device for height-limited objects applied to vehicles provided in this application. Figure 5 As shown, the detection device 50 for height-restricted objects applied to vehicles provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0198] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0199] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0200] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0201] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk storage.

[0202] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0203] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0204] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0205] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0206] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0207] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0208] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0210] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0211] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0212] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for detecting a height-restricted object applied to a vehicle, characterized in that: include: Acquire an image and initial point cloud data of a driving front of the vehicle, and determine a bounding box of a height-restricted object in the image; Determining point cloud data of the height-limited object from the initial point cloud data according to the bounding box; A detection result indicating whether the vehicle can pass through the height-restricted object is generated according to the point cloud data of the height-restricted object and the height information of the vehicle.

2. The method according to claim 1, characterized in that Determining point cloud data of the height-limited object from the initial point cloud data according to the bounding box includes: Converting the two-dimensional coordinates of the bounding box to the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box; Based on the point cloud coordinates of the bounding box, point cloud data outside the bounding box is filtered out from the initial point cloud data to obtain point cloud data of the height-limited object.

3. The method according to claim 2, characterized in that Converting the two-dimensional coordinates of the bounding box to the three-dimensional coordinate system corresponding to the initial point cloud data to obtain the point cloud coordinates of the bounding box includes: The two-dimensional coordinates of the bounding box are calculated and processed according to a preset external parameter matrix and a preset internal parameter matrix to obtain the point cloud coordinates of the bounding box.

4. The method according to claim 2, characterized in that: The two-dimensional coordinates of the bounding box include the two-dimensional coordinates of two corner points on a diagonal line of the bounding box.

5. The method according to claim 1, characterized in that Generating a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle, including: Performing clustering processing on the point cloud data of the height-limited object to obtain a three-dimensional contour point cloud set of the height-limited object; The detection result is generated according to the three-dimensional contour point cloud set and the height information of the vehicle.

6. The method according to claim 5, characterized in that Generating the detection result according to the three-dimensional contour point cloud set and the height information of the vehicle includes: Determine the three-dimensional contour point cloud with the smallest height in the three-dimensional contour point cloud set as the first point cloud of the height-limited object; From the first point clouds of the height-limited objects, determine the first point cloud closest to the vehicle as the second point cloud; The detection result is generated according to the second point cloud and the height information of the vehicle.

7. The method according to claim 6, characterized in that Generating the detection result according to the second point cloud and the height information of the vehicle includes: If it is determined that the z-axis coordinate value of the second point cloud is greater than the z-axis coordinate value in the height information, then determining that the detection result is that the vehicle can pass through the height-restricted object; If it is determined that the z-axis coordinate value of the second point cloud is less than or equal to the z-axis coordinate value in the height information, then the detection result is determined to be that the vehicle cannot pass through the height-restricted object.

8. The method according to any one of claims 1 to 7, characterized in that Obtain the initial point cloud data in front of the vehicle, including: The initial point cloud data is collected by the 4D millimeter wave radar on the vehicle.

9. The method according to any one of claims 1 to 7, characterized in that: Determining a bounding box of a height-limited object in the image includes: Inputting the image into a preset deep learning model to extract features and obtain image feature data; The image feature data is recognized and processed based on the preset deep learning model to obtain a bounding box of the height-limited object.

10. The method according to any one of claims 1 to 7, characterized in that The height-restricted object has a category; the method further includes: Generate and issue a prompt message, wherein the prompt message includes the category of the height-restricted object and the detection result.

11. A detection device for height-limited objects applied to vehicles, characterized in that: include: An acquisition unit, used for acquiring an image and initial point cloud data in front of the vehicle; A first determining unit, configured to determine a bounding box of a height-limited object in the image; A second determining unit, configured to determine the point cloud data of the height-limited object from the initial point cloud data according to the bounding box; A generating unit is used to generate a detection result indicating whether the vehicle can pass through the height-restricted object according to the point cloud data of the height-restricted object and the height information of the vehicle.

12. A detection device for height-limited objects applied to vehicles, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images