Vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis

Abstract

The application relates to the technical field of vehicle obstacle avoidance, in particular to a vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis, which comprises the following steps: acquiring road images, millimeter wave point clouds and laser point clouds in front of a vehicle in real time; segmenting laser point clouds at each moment into multiple structure surfaces, acquiring road surfaces in the laser point clouds according to the areas and heights of the structure surfaces, and further acquiring each obstacle in the laser point clouds; acquiring each extension of each obstacle in the millimeter wave point clouds; acquiring the self extension index of each extension at each moment according to the distance and point cloud density of each extension at each moment, combining the area of each connected domain in the road image to acquire the point cloud extension index of each extension at each moment, further calculating the window size of each extension at each moment, extending each extension at each moment, and further detecting obstacles in front of the industrial vehicle. The application adaptively acquires the window size in the extension process, and improves the vehicle obstacle avoidance effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle obstacle avoidance, in particular to a vehicle obstacle avoidance method based on multi-modal data analysis of industrial vehicle driving. BACKGROUND

[0002] The vehicle advanced driving assistance system (ADAS) has been widely applied in the safety control field of industrial vehicles. The early ADAS detects obstacles around the industrial vehicle through visual detection, and brakes in time when detecting obstacles within a certain distance, so as to realize vehicle obstacle avoidance. However, the visual detection method cannot accurately reflect the distance between the vehicle and the obstacle, the moving speed and direction of the dynamic obstacle, which may cause misjudgment and misstop during obstacle avoidance, thereby affecting the safety and transportation efficiency of the vehicle. Therefore, it is of great significance to construct an industrial vehicle obstacle avoidance method combining multi-modal data for improving the safety of workers and vehicles.

[0003] The prior art usually uses the fusion of visual detection and millimeter wave radar detection to improve the determination of the position of the obstacle by the vehicle assistance system. However, the point cloud data collected by the millimeter wave radar is sparse, and if it is directly fused, there will be a large error, so the point cloud data needs to be expanded. The prior art expands the point cloud data collected by the millimeter wave radar using a fixed window size, and does not fully consider that in the industrial transportation scene, the stacking density of goods and the distance between the vehicle and the goods will dynamically change, so the scanned point cloud data at different times and different distances will also dynamically change. Since the fixed expansion method cannot adapt to the dynamic changes of the point cloud density, the expansion effect of the point cloud data will be affected, thereby reducing the fusion precision of the cross-modal data and affecting the vehicle obstacle avoidance effect. SUMMARY

[0004] In order to solve the above technical problems, the present application provides a vehicle obstacle avoidance method based on multi-modal data analysis of industrial vehicle driving to solve the existing problems.

[0005] The vehicle obstacle avoidance method based on multi-modal data analysis of industrial vehicle driving of the present application adopts the following technical scheme:

[0006] One embodiment of the present application provides a vehicle obstacle avoidance method based on multi-modal data analysis of industrial vehicle driving, which comprises the following steps:

[0007] real-time acquisition of road images, millimeter wave point clouds and laser point clouds in front of the industrial vehicle;

[0008] The road images, the millimeter wave point cloud and the laser point cloud at each time are spatially aligned; the laser point cloud at each time is segmented into a plurality of structure surfaces, the road surface confidence of each structure surface is obtained according to the area and height of each structure surface, and then the road surface in the laser point cloud at each time is obtained; the laser point cloud data at each time is segmented into a plurality of objects, and each obstacle in the laser point cloud at each time is obtained according to the overlapping degree between each object and the road surface at the same time in the two-dimensional plane.

[0009] Each extended object in the millimeter wave point cloud at each time is obtained according to the spatial position of each obstacle in the laser point cloud at each time in the millimeter wave point cloud; and the self expansion index of each extended object at each time is obtained according to the distance between each extended object and the industrial vehicle at each time and the point cloud density inside each extended object.

[0010] The road image at each time is segmented into a plurality of connected domains, the point cloud expansion index of each extended object at each time is obtained according to the area of the connected domain with the highest overlapping degree with each extended object in the road image at each time and the self expansion index of each extended object, and then the preset initial window size of each extended object at each time is adjusted to obtain the extended extended object at each time, and the obstacle in front of the industrial vehicle is detected in combination with the road image at each time.

[0011] Preferably, the calculation formula of the road surface confidence of each structure surface is: ; in the formula, is the road surface confidence of the rth structure surface; is the area of the rth structure surface; is the maximum value in the area of all structure surfaces at the time when the rth structure surface is located; is the mean value of the z coordinates of all point clouds inside the rth structure surface; is a preset constant; wherein the area of each structure surface refers to the total number of point clouds in each structure surface.

[0012] Preferably, the road surface in the laser point cloud at each time refers to the structure surface with the maximum road surface confidence in the laser point cloud at each time.

[0013] Preferably, the method for obtaining each obstacle in the laser point cloud at each time is: respectively performing plane projection on each object and the road surface in the laser point cloud at each time; obtaining the bounding box of the road surface and each object in the projection plane through the AABB bounding box algorithm; taking the bounding box of the road surface and each object as the input of the calOverTotal method, and sequentially outputting the intersection value between the road surface and each object; recording each object corresponding to the positive number of the intersection value in the laser point cloud at each time as each obstacle in the laser point cloud at each time.

[0014] Preferably, the method for obtaining each extension in the millimeter wave point cloud at each time is as follows: in the laser point cloud data at each time, the maximum and minimum values of the coordinates of each obstacle in each coordinate axis direction are obtained respectively; in the millimeter wave point cloud data corresponding to the same time, if the x, y and z coordinate values of a point cloud are simultaneously between the minimum and maximum values of the corresponding axis direction of an obstacle, then the point cloud is recorded as the corresponding obstacle point of the obstacle in the millimeter wave point cloud; and the object composed of all the corresponding obstacle points of each obstacle in the millimeter wave point cloud at each time is recorded as each extension in the millimeter wave point cloud at each time.

[0015] Preferably, the method for obtaining the self extension index of each extension at each time is as follows: the bounding box of each extension in the three-dimensional space at each time is obtained; the ratio of the number of internal point clouds of each extension to the volume of the bounding box of each extension is recorded as the first density of each extension at each time; the mean value of the target distances of all the point clouds inside each extension at each time is calculated; and the ratio of the mean value to the first density is recorded as the self extension index of each extension at each time.

[0016] Preferably, the method for obtaining the point cloud extension index of each extension at each time is as follows: all the connected domains in the road image at each time are obtained; the connected domain with the highest overlap degree with each extension at the corresponding time is recorded as the connected domain of the corresponding extension; the area of the minimum circumscribed rectangle of the connected domain corresponding to each extension at each time is obtained; and the ratio of the self extension index to the area is recorded as the point cloud extension index of each extension at each time.

[0017] Preferably, the calculation formula for adjusting the preset initial window size of each extension at each time is as follows: ; in the formula, is the adjusted window size of the u-th extension at the i-th time; is the preset initial window size; is the preset adjustment coefficient; tanh() is the hyperbolic tangent function; is the point cloud extension index of the u-th extension at the i-th time.

[0018] Preferably, the specific process of each extension after expansion at each time is as follows: all the point clouds inside each extension at each time are respectively taken as the input of the KNN neural network, the calculated adjusted window size of each extension at each time is taken as the window size of each point cloud inside the corresponding extension, and each extension after expansion at each time is output.

[0019] Preferably, the specific process of detecting the obstacles in front of the industrial vehicle is: fusing the extended millimeter wave point cloud at each time with the road image to obtain a depth image; inputting the depth image into a vehicle auxiliary system, detecting the obstacles through a visual detection method, and obtaining the distance and direction between the obstacles and the vehicle through a millimeter wave radar ranging method, so as to evaluate the position and direction of the obstacles at each time.

[0020] The present application has at least the following beneficial effects:

[0021] The present application is aimed at the problem that the prior art does not fully consider that the point cloud density will dynamically change when extending the millimeter wave point cloud data, and the fixed window extension method has poor adaptability, resulting in poor point cloud extension effect. The present application can accurately distinguish the obstacles in the laser point cloud by recognizing the road surface, thereby providing support for subsequent adaptive extension of the millimeter wave point cloud corresponding to different obstacles; by analyzing the point cloud density characteristics of the distance of each extended object from the vehicle, the self-extension index is constructed, thereby the extension degree required by each extended object in the millimeter wave point cloud can be preliminarily evaluated; then, combined with the area of the corresponding connected domain of each extended object in the road image, the point cloud extension index is constructed, thereby the extension degree can be more accurately determined in combination with the visual image, the extension effect of the millimeter wave point cloud is improved, and the fusion accuracy of the cross-modal data is improved, thereby the obstacle avoidance effect of the industrial vehicle is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0023] Figure 1 The step flow chart of the vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis provided by the present application;

[0024] Figure 2 The flow chart of obtaining the point cloud extension index of each extended object at each time provided by the present application. DETAILED DESCRIPTION

[0025] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined object, the specific implementation, structure, features and effects of the vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis according to the present application are described in detail below in combination with the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0027] The specific scheme of the vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis provided by the present application is described below in combination with the drawings.

[0028] The vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis provided by one embodiment of the present application specifically provides the following vehicle obstacle avoidance method based on industrial vehicle driving multi-modal data analysis, please refer to Figure 1 The method comprises the following steps:

[0029] Step one: real-time acquisition of road images, millimeter wave point clouds and laser point clouds in front of the industrial vehicle.

[0030] The present application analyzes the obstacle avoidance mode in the forward direction of the industrial vehicle. The high-definition camera in the ADAS system collects real-time road image data in front of the industrial vehicle; the millimeter wave radar and the laser radar collect real-time road point cloud data in front of the industrial vehicle, and the collected point cloud data is respectively recorded as millimeter wave point cloud and laser point cloud. The collection frequency of the camera, the millimeter wave radar and the laser radar is 10Hz.

[0031] The collected road images and point cloud data at each time are filtered by a filtering algorithm. The filtering algorithm is not limited to median filtering and Gaussian filtering, and median filtering is used in this embodiment.

[0032] Step two: spatial alignment of road images, millimeter wave point clouds and laser point clouds at each time; segmentation of laser point clouds at each time into multiple structure surfaces, acquisition of road surface confidence of each structure surface according to the area and height of each structure surface, and further acquisition of road surface in laser point clouds at each time; segmentation of laser point cloud data at each time into multiple objects, and acquisition of each obstacle in laser point clouds at each time according to the overlapping degree between each object and the projection of the road surface at the same time in the two-dimensional plane.

[0033] Since the point cloud data collected by the millimeter wave radar has the problem of sparse point cloud density, it is usually necessary to expand the millimeter wave point cloud to generate more dense point cloud data, so as to ensure that the millimeter wave point cloud data can be accurately aligned in space with the visual image. However, in the actual industrial transportation environment, when the industrial vehicle is transporting goods, the detection radar will usually capture a large number of targets, such as goods piled up on the driving road, shelves on both sides of the road, and industrial vehicles driving in front.

[0034] Among all the captured targets, part of them are static or dynamic obstacles that need to be avoided, but there are also some backgrounds or non-obstacles. If all the point cloud data is expanded, redundant information may be introduced, which will affect the efficiency and accuracy of the cross-modal fusion of point cloud data and image data. Therefore, it is necessary to screen out the point cloud data that really belongs to the obstacle from all the millimeter wave point cloud data, and to adaptively adjust the expansion range by analyzing the spatial distribution characteristics of the obstacle point cloud, so as to improve the cross-modal feature matching accuracy of the millimeter wave point cloud data and the image data, and to improve the obstacle avoidance effect of the industrial vehicle.

[0035] Taking the i-th moment as an example, the road image, millimeter wave point cloud and laser point cloud are spatially aligned. The alignment method is not limited to the chessboard calibration method and two-step method. The spatial alignment of the road image, millimeter wave radar and laser radar is a known technology, which will not be described here. Since the millimeter wave point cloud data is relatively sparse, there will be a large error when distinguishing obstacles; and since it has been spatially aligned with the laser point cloud, the laser point cloud can be used to screen obstacles, and the obstacle point cloud can be located in the millimeter wave point cloud according to its spatial position.

[0036] The laser point cloud data collected at the i-th moment is taken as the input of the structure surface segmentation method, so as to identify a plurality of structure surfaces in the laser point cloud data. Since the density of the laser point cloud is relatively high, the point cloud density of each structure surface can be approximately regarded as uniform, so the total number of point clouds in each structure surface is recorded as the area of each structure surface. The structure surface segmentation method is not limited to the RANSAC algorithm and the RegionGrowing method of the PCL point cloud library.

[0037] Since the vehicle is driving forward, the objects on the road surface in front must be obstacles, so the road surface can be located and the relationship between each object and the road surface can be analyzed to extract the obstacles. Since the height of the road surface is the lowest and the area is usually the largest compared to the structure surface formed by other objects.

[0038] As a preferred embodiment, the road surface confidence of each structure surface is obtained according to the area and height of each structure surface, which is used to represent the possibility of each structure surface being a road surface.

[0039] In this embodiment, the road surface confidence score of the r-th structural surface is denoted as... Its specific expression is: In the formula, Let be the pavement confidence level of the r-th structural surface; Let r be the area of ​​the r-th structural surface; The maximum area among all structural surfaces at the time of the r-th structural surface is obtained by... Come to Unitize to avoid The problem arises when the value is too large, leading to a situation where other parameters have too little impact. Let z be the mean z-coordinate of all point clouds inside the r-th structural surface; This is a preset constant. To avoid the denominator being 0, its value range is (0.005, 0.01). The value has little impact on the calculation and can be ignored. In this embodiment, it is 0.008.

[0040] The road surface confidence score reflects the probability that each structural surface is a road surface; the higher the score, the more likely the corresponding structural surface is to be a road surface.

[0041] Calculate the road surface confidence of all structural surfaces in the laser point cloud at time i, and denote the structural surface corresponding to the maximum value among all road surface confidence as the road surface at time i.

[0042] Furthermore, considering that in industrial transportation scenarios, there may be obstacles that do not directly contact the road surface, such as pallets extending from shelves, high beams, and hoisting equipment, although these obstacles do not contact the road surface, they are still within the space where the vehicle travels, so collisions may still occur, which requires further analysis.

[0043] The laser point cloud data at time i is used as input to the PointNet network, which segments the laser point cloud into multiple objects. The laser point cloud data of each object in the laser point cloud at time i, as well as the laser point cloud data of the road surface, are then subjected to planar projection, i.e., orthogonally projected along the z-axis onto a two-dimensional plane with xy coordinates, to eliminate the influence of z-axis height.

[0044] In a two-dimensional plane, obtain the maximum values ​​of the laser point cloud inside the road surface in the x and y coordinates. , minimum coordinates , The bounding box of the road surface in the 2D plane is obtained using the AABB bounding box algorithm. The bounding boxes of each object in the laser point cloud are obtained in the same way.

[0045] The bounding boxes of the road surface and each object are sequentially input into the calOverTotal method, and the calOverTotal method sequentially outputs the intersection values between the road surface and each of the other objects. When the intersection value is positive, it means that the road surface can contain the object or intersect with the bounding box profile of the object, so as to reflect that if the height influence is not considered, the object will overlap the road surface, and therefore the object is determined as an obstacle.

[0046] The object corresponding to the positive intersection value in the laser point cloud at the ith moment is recorded as the obstacle in the laser point cloud at the ith moment. At this point, the obstacle that can possibly collide with the current vehicle at the ith moment in the advancing direction can be preliminarily extracted.

[0047] Step three: according to the spatial positions of each obstacle in the laser point cloud at each moment in the millimeter wave point cloud, each extended object in the millimeter wave point cloud at each moment is obtained; and according to the distance between each extended object and the industrial vehicle at each moment and the point cloud density inside each extended object, the self-expansion index of each extended object at each moment is obtained.

[0048] After the obstacles are extracted, the spatial distribution characteristics of each obstacle in the millimeter wave point cloud can be analyzed to further evaluate the required expansion degree. For obstacles that are relatively close and have a high millimeter wave point cloud density, the shape perception of such obstacles is already relatively complete, and if a large degree of expansion is performed, neighborhood noise will be easily introduced, which will instead generate additional errors, and therefore the expansion degree of such obstacles should be small. For obstacles that are relatively far away and have a sparse millimeter wave point cloud density, since the point cloud is sparse and cannot completely reflect the geometric characteristics, a large degree of expansion is required to compensate for the information loss, so as to ensure the matching accuracy of the millimeter wave point cloud and the image and provide effective spatial information.

[0049] Taking the u-th obstacle in the laser point cloud at the ith moment as an example for analysis. In the laser point cloud data, the maximum and minimum values of the coordinates of the u-th obstacle in each coordinate axis direction are obtained; in the millimeter wave point cloud data at the same moment, if the x, y, and z coordinate values of a point are simultaneously between the minimum and maximum values of the corresponding axis direction of the u-th obstacle, the point is recorded as the corresponding obstacle point of the u-th obstacle in the millimeter wave point cloud. Through this cross-modal alignment method, the problem of being unable to accurately distinguish objects due to the sparsity of the millimeter wave radar collected point cloud data can be solved.

[0050] The object composed of all the corresponding obstacle points of the u-th obstacle in the millimeter wave point cloud at the ith moment is recorded as the u-th extended object in the millimeter wave point cloud at the ith moment.

[0051] By the above construction method of the bounding box, the bounding box of the u-th extended object in the three-dimensional space is obtained by the AABB algorithm, and the volume of the bounding box is obtained; then the ratio of the number of internal point clouds of the u-th extended object to the volume of the bounding box is recorded as the first density of the u-th extended object. The first density can reflect the point cloud distribution density of the u-th extended object in the three-dimensional space; the greater the value, the greater the density, and the smaller the degree of expansion required. The calculation of the volume of the bounding box is a known technology, which will not be described here.

[0052] In the collected millimeter wave point cloud data, the data form is the target distance, target angle, target speed and point cloud coordinates of each point cloud. Among them, the target distance of each point cloud refers to the distance between each point cloud and the millimeter wave radar. The average of the target distances of all point clouds inside the u-th extended object is calculated, and the obtained average can reflect the distance between the u-th extended object and the industrial vehicle, and the smaller the value, the closer the distance, and the less the need for large expansion.

[0053] As a preferred embodiment, according to the distance between each extended object and the industrial vehicle at each time, and the point cloud density inside each extended object, the self-expansion index of each extended object at each time is obtained, which is used to represent the degree of expansion required by each extended object at each time.

[0054] In this embodiment, the self-expansion index of the u-th extended object at the i-th time is recorded as , and its specific expression is: ; in the formula, is the self-expansion index of the u-th extended object at the i-th time; is the average of the target distances of all point clouds inside the u-th extended object at the i-th time; is the first density of the u-th extended object at the i-th time.

[0055] The self-expansion index can reflect the expansion degree of the u-th extended object in the millimeter wave point cloud; the smaller the value, the closer the distance between the u-th extended object and the vehicle, the denser the point cloud density, and the smaller the degree of expansion required, so as to avoid introducing noise and affecting the distance and moving direction evaluation of the vehicle to the obstacle, and affecting the obstacle avoidance effect.

[0056] Step four: dividing the road image at each time into multiple connected domains, obtaining the point cloud expansion index of each extended object at each time according to the area of the connected domain with the highest overlap degree with each extended object in the road image at each time, and the self-expansion index of each extended object, and then adjusting the preset initial window size of each extended object at each time, obtaining the expanded each extended object at each time, and combining the road image at each time to detect the obstacle in front of the industrial vehicle.

[0057] Further, although the extension degree of the object in the millimeter wave point cloud can be preliminarily evaluated by the self extension index, errors may occur in the millimeter wave radar due to the influence of multipath reflection or ground clutter when collecting the point cloud data, thereby causing the distance and density calculation of the point cloud data to deviate. Therefore, in order to obtain a more accurate extension degree, multi-modal analysis can be combined with the visual image.

[0058] Since the road image and the millimeter wave point cloud have been spatially aligned, that is, the three-dimensional point cloud data is projected into the two-dimensional road image, the road image at the i th time is converted into a gray image, and all connected domains in the gray image are obtained through a connected domain generation algorithm. Among all the connected domains, the connected domain with the highest overlap degree with the u th extended object is recorded as the connected domain of the u th extended object. The connected domain generation algorithm is not limited to the region growing algorithm, seed filling method.

[0059] In the visual image, the size of the object is also affected by the distance between the object and the camera. The closer the distance between the object of the same size and the camera, the larger the area in the image. When the connected domain area corresponding to the u th extended object is large, if the self extension index of the u th extended object is small, it indicates that the point cloud in the extended object is relatively dense, and there is no need to expand it to a large extent; if the self extension index of the u th extended object is large, it indicates that the point cloud in the extended object is more sparse, which does not conform to the normal characteristics, and needs to be expanded to a large extent. In addition, the millimeter wave radar cannot scan the point cloud at a long distance well, so any extended object at a long distance, that is, the connected domain with a small area, needs to be expanded to a large extent.

[0060] In the road image at the i th time, the area of the minimum circumscribed rectangle of the connected domain corresponding to the u th extended object is obtained. The obtained area can reflect the area of the connected domain corresponding to the u th extended object in the image, and the larger the value, the larger the area, and the closer the distance between the u th obstacle and the vehicle.

[0061] As a preferred embodiment, according to the area of the connected domain with the highest overlap degree of each extended object in the road image at each time and the self extension index of each extended object, the point cloud extension index of each extended object at each time is obtained, which is used to represent the extension degree of each extended object at each time. The flow chart for obtaining the point cloud extension index of each extended object at each time is shown in FIG. 2. Figure 2

[0062] In this embodiment, the point cloud extension index of the u th extended object at the i th time is denoted as , and the specific expression is: ; in the formula, is the point cloud extension index of the u th extended object at the i th time, ​is the self-expansion index of the u-th extended object at the i-th moment, is the area of the minimum circumscribed rectangle of the connected domain corresponding to the u-th extended object at the i-th moment.

[0063] The point cloud expansion index can combine the feature information of the millimeter wave radar and the visual image, thereby comprehensively reflecting the expansion degree of the u-th extended object in the millimeter wave point cloud. The smaller the point cloud expansion index, the larger the area of the connected domain corresponding to the u-th extended object, and the smaller the self-expansion index value of the extended object, the denser the point cloud in the extended object, and the less the extended object needs to be expanded to avoid errors. When the area of the connected domain corresponding to the u-th extended object is small, it means that the u-th extended object is far away from the current vehicle, and the self-expansion index corresponding to the extended object will also be larger accordingly, and the calculated point cloud expansion index will also be larger, which means that the extended object should be expanded to a larger extent.

[0064] It should be noted that if the number of point clouds inside the u-th extended object is 0, the point cloud expansion index of the u-th extended object is set to 0, indicating that no expansion is performed to avoid errors.

[0065] Further, in the embodiment, the KNN neural network model is used to expand the point clouds in each extended object. In the expansion process of the KNN neural network, a corresponding window needs to be set for each point cloud, and the window size determines the expansion degree of the point cloud. The larger the window, the greater the expansion degree of the point cloud; on the contrary, the smaller the expansion degree of the point cloud.

[0066] Taking the u-th extended object at the i-th moment as an example, the adjusted window size of the u-th extended object at the i-th moment is calculated, and the specific expression is: ; in the formula, is the adjusted window size of the u-th extended object at the i-th moment; is a preset initial window size, which is taken as 0.2m in the embodiment, and m is the length unit meter; is a preset adjustment coefficient, in order to avoid the problem of introducing additional noise caused by excessive parameter adjustment, the value is taken between 0.2-0.35, and the value is taken as 0.3 in the embodiment; tanh() is a hyperbolic tangent function, which is used for normalizing the input data; is the point cloud expansion index of the u-th extended object at the i-th moment.

[0067] All point clouds in the u-th extended object at the i-th moment are taken as the input of the KNN neural network, the adjusted window size of the u-th extended object at the i-th moment is taken as the window size of each point cloud inside the u-th extended object, and the u-th extended object after expansion at the i-th moment is output.

[0068] Similarly, the point cloud expansion index of all extended objects in the millimeter wave point cloud at the i th moment is calculated, so that the window size of each extended object can be adaptively calculated, and then the point cloud inside all extended objects is expanded, so that the expansion degree of the near dense target is smaller to avoid introducing noise, and the expansion degree of the far sparse target is larger to supplement information, thereby improving the expansion effect of the point cloud data.

[0069] Further, the millimeter wave point cloud after expansion at the i th moment is fused with the road image, so as to obtain a depth image. The depth image is input into a vehicle auxiliary system, the obstacle is detected by a visual detection method, and the distance and direction between the obstacle and the vehicle are obtained by a millimeter wave radar ranging method, so that the position and direction of the obstacle at the moment can be more accurately evaluated, and the problem that the vehicle stops to avoid the obstacle but the vehicle collides with the dynamic obstacle is avoided. Similarly, the vehicle at each moment is avoided. Through the feature fusion of multi-modal data, the obstacle avoidance effect of the industrial vehicle can be improved, and the efficiency and safety of transportation are further improved.

[0070] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. Moreover, the above-mentioned embodiments of the present application are described. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or can be advantageous.

[0071] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments.

[0072] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; the technical solutions described in the above embodiments are modified, or some technical features are replaced, without changing the essence of the corresponding technical solutions out of the scope of the technical solutions of the embodiments of the present application, which should be included in the protection scope of the present application.

Claims

1. A vehicle obstacle avoidance method based on multimodal data analysis of industrial vehicle driving, characterized in that, The method includes the following steps: Real-time acquisition of road images, millimeter-wave point clouds, and laser point clouds in front of industrial vehicles; The road images, millimeter-wave point clouds, and laser point clouds at each time moment are spatially aligned; the laser point clouds at each time moment are segmented into multiple structural surfaces, and the road surface confidence of each structural surface is obtained based on the area and height of each structural surface, thereby obtaining the road surface in the laser point cloud at each time moment; the laser point cloud data at each time moment is segmented into multiple objects, and each obstacle in the laser point cloud at each time moment is obtained based on the degree of overlap between the projection of each object and the road surface at the same time moment on the two-dimensional plane. Based on the spatial position of each obstacle in the laser point cloud at each time point in the millimeter-wave point cloud, each extended object in the millimeter-wave point cloud at each time point is obtained; based on the distance between each extended object and the industrial vehicle at each time point, and the point cloud density inside each extended object, the self-expansion index of each extended object at each time point is obtained. The road image at each time point is segmented into multiple connected regions. Based on the area of ​​the connected region with the highest degree of overlap with each extension in the road image at each time point, and the extension index of each extension itself, the point cloud extension index of each extension at each time point is obtained. Then, the preset initial window size of each extension at each time point is adjusted to obtain each extension after expansion at each time point. Combined with the road image at each time point, obstacles in front of the industrial vehicle are detected.

2. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The formula for calculating the pavement confidence level of each structural surface is as follows: In the formula, Let be the pavement confidence level of the r-th structural surface; Let r be the area of ​​the r-th structural surface; It is the maximum area among all structural surfaces at the time when the r-th structural surface is located; Let z be the mean z-coordinate of all point clouds inside the r-th structural surface; This is a preset constant; where the area of ​​each structural surface refers to the total number of point clouds within each structural surface.

3. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The road surface in the laser point cloud at each time point refers to the structural surface with the highest road surface confidence in the laser point cloud at each time point.

4. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The method for obtaining each obstacle in the laser point cloud at each time moment is as follows: each object and the road surface in the laser point cloud at each time moment are projected onto a plane; the bounding boxes of the road surface and each object in the projection plane are obtained by using the AABB bounding box algorithm; the bounding boxes of the road surface and each object are used as inputs to the calOverTotal method, and the overlap values ​​between the road surface and each object are output sequentially; each object with a positive overlap value in the laser point cloud at each time moment is recorded as an obstacle in the laser point cloud at each time moment.

5. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The method for obtaining each extension in the millimeter-wave point cloud at each time moment is as follows: In the laser point cloud data at each time moment, the maximum and minimum coordinate values ​​of each obstacle in each coordinate axis direction are obtained respectively; In the millimeter-wave point cloud data corresponding to the same time moment, if the x, y, z coordinate values ​​of a point cloud are simultaneously between the minimum and maximum values ​​of the corresponding axis of a certain obstacle, then the point cloud is recorded as the corresponding obstacle point of the obstacle in the millimeter-wave point cloud; The objects composed of all the corresponding obstacle points of each obstacle in the millimeter-wave point cloud at each time moment are recorded as each extension in the millimeter-wave point cloud at each time moment.

6. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The method for obtaining the self-expansion index of each extension at each time step is as follows: obtain the bounding box of each extension at each time step in three-dimensional space; record the ratio of the number of internal point clouds of each extension to the volume of its bounding box as the first density of each extension at each time step; calculate the average target distance of all point clouds inside each extension at each time step; and record the ratio of the average value to the first density as the self-expansion index of each extension at each time step.

7. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The method for obtaining the point cloud expansion index of each extended object at each time point is as follows: obtain all connected components in the road image at each time point; record the connected component with the highest degree of overlap with each extended object at the corresponding time point as the connected component of each extended object; obtain the area of ​​the minimum bounding rectangle of the connected component corresponding to each extended object at each time point; and record the ratio of the self-expansion index to the area as the point cloud expansion index of each extended object at each time point.

8. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The calculation formula for adjusting the preset initial window size of each extension at each time step is as follows: In the formula, The adjusted window size for the u-th extension at time i; Set the initial window size; tanh() is the preset adjustment coefficient; tanh() is the hyperbolic tangent function; Let be the point cloud expansion index of the u-th extended object at the i-th time.

9. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The specific process for each extended object at each time step is as follows: take all point clouds in each extended object at each time step as input to the KNN neural network, take the calculated adjusted window size of each extended object at each time step as the window size of each point cloud in the corresponding extended object, and output each extended object at each time step.

10. The vehicle obstacle avoidance method based on multimodal driving data analysis of industrial vehicles as described in claim 1, characterized in that, The specific process for detecting obstacles in front of industrial vehicles is as follows: the expanded millimeter-wave point cloud at each time moment is fused with the road image to obtain a depth image; the depth image is input into the vehicle assistance system, obstacles are detected by visual detection, and the distance and direction between the obstacle and the vehicle are obtained by millimeter-wave radar ranging, thereby evaluating the position and direction of the obstacle at each time moment.

Images