Method and device for detecting road potholes, processor and vehicle

Abstract

The embodiment of the application provides a kind of for detecting road surface pit method and device, processor and vehicle, belong to vehicle technical field.Vehicle is provided with image acquisition device and point cloud data acquisition device, for detecting road surface pit method includes: obtaining the color image and point cloud data corresponding to the road surface to be detected that image acquisition device and point cloud data acquisition device are collected simultaneously respectively;The position information of the circumscribed rectangular region corresponding to pit is obtained by detecting color image through pre-trained pit detection model, wherein, pit detection model is improved YOLOv5 model, and C3 feature extraction module in the backbone network of improved YOLOv5 model is replaced by light weight convolutional neural network;According to the point cloud data corresponding to position information and the point cloud data corresponding to the peripheral region of circumscribed rectangular region, the depth of pit is determined;According to the point cloud data corresponding to position information, the length and width of pit are determined.The embodiment of the application can accurately detect pit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a method and device for detecting road pits, a processor and a vehicle. BACKGROUND

[0002] During the driving of a vehicle, if the road is uneven, it may cause the vehicle to drive with difficulty, and if a deep pit is encountered on the driving route of the vehicle, the vehicle may be unable to drive normally, and even more seriously, the safety of the personnel in the vehicle may be affected. In order to detect the pit on the driving route, a distance measuring sensor is usually arranged on the vehicle at present, and whether there is a pit in front of the vehicle is determined by the distance change of the road on the driving route. However, the pit detection by the distance measuring sensor is affected by the distance between the vehicle and the pit, and the depth and other size information of the pit cannot be accurately detected when the vehicle is far away from the pit, so a more accurate pit detection method is needed. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a method and device for detecting road pits, a method and device for early warning of road pits, a processor, a vehicle and a storage medium, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above-mentioned purpose, the first aspect of the embodiments of the present application provides a method for detecting road pits, applied to a vehicle, the vehicle being provided with an image acquisition device and a point cloud data acquisition device, the method comprising:

[0005] acquiring a color image and point cloud data corresponding to a to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device at the same time respectively;

[0006] detecting the color image by using a pre-trained pit detection model to obtain position information of an external rectangular region corresponding to the pit, wherein the pit detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network;

[0007] determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral region of the external rectangular region;

[0008] determining the length and width of the pit according to the point cloud data corresponding to the position information.

[0009] In the embodiments of the present application, the lightweight convolutional neural network comprises an ECA attention mechanism.

[0010] In the embodiments of the present application, a path aggregation network in a neck network of the improved YOLOv5 model is replaced by a bidirectional feature pyramid network.

[0011] In the embodiment of the present application, the position information comprises corner point position information and center position information; the pre-trained pit detection model is used to detect the color image to obtain the position information of the circumscribed rectangular region corresponding to the pit, comprising: the pre-trained pit detection model is used to detect the color image to obtain the corner point position information of the circumscribed rectangular region corresponding to the pit; and the center position information of the circumscribed rectangular region is determined according to the corner point position information.

[0012] In the embodiment of the present application, the depth of the pit is determined according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, comprising: a first height value in the point cloud data corresponding to the center position information and a second height value in the point cloud data corresponding to the peripheral region of the circumscribed rectangular region are obtained; and the absolute value of the difference between the first height value and the second height value is determined to obtain the depth of the pit.

[0013] The second aspect of the embodiment of the present application provides a method for road surface pit early warning, applied to a vehicle, comprising:

[0014] Obtaining size information of the pit, wherein the size information comprises depth, length and width, and the size information is determined by the method for detecting the road surface pit described above;

[0015] Determining that the type of the pit is a non-passable pit type according to the size information and a preset size threshold;

[0016] Determining the distance between the pit and the vehicle according to the point cloud data corresponding to the position information;

[0017] Determining a corresponding target early warning strategy according to the distance;

[0018] Controlling the engineering machinery to perform a corresponding early warning action according to the target early warning strategy.

[0019] In the embodiment of the present application, the type of the pit is determined to be a non-passable pit type according to the size information and a preset size threshold, comprising: in the case that the depth, length and width of the pit are all greater than the corresponding preset size threshold, the type of the pit is determined to be a non-passable pit type.

[0020] In the embodiment of the present application, the corresponding target early warning strategy is determined according to the distance, comprising: determining the distance interval where the distance is located; and determining the early warning strategy corresponding to the distance interval as the target early warning strategy.

[0021] In the embodiment of the present application, the distance interval includes a first distance interval, a second distance interval and a third distance interval from far to near, the pre-warning strategy corresponding to the first distance interval includes issuing prompt information indicating that there is an impassable pit in the front road surface, the pre-warning strategy corresponding to the second distance interval includes issuing prompt information indicating that the vehicle has approached the impassable pit in the front road surface and needs to change the driving track of the vehicle, and the pre-warning strategy corresponding to the third distance interval includes controlling the vehicle to slow down or stop.

[0022] The third aspect of the embodiment of the present application provides a processor configured to execute the method for detecting a road surface pit according to the above-mentioned embodiment or the method for road surface pit pre-warning according to the above-mentioned embodiment.

[0023] The fourth aspect of the embodiment of the present application provides a device for detecting a road surface pit, applied to a vehicle, wherein the vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the device comprises:

[0024] A data acquisition module is configured to acquire a color image and point cloud data corresponding to a to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device respectively at the same time;

[0025] A pit detection module is configured to detect the color image by using a pre-trained pit detection model to obtain position information of an external rectangular region corresponding to a pit; wherein the pit detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network;

[0026] A size determination module is configured to determine the depth of the pit according to point cloud data corresponding to the position information and point cloud data corresponding to a peripheral region of the external rectangular region, and determine the length and width of the pit according to the point cloud data corresponding to the position information.

[0027] The fifth aspect of the embodiment of the present application provides a device for road surface pit pre-warning, applied to a vehicle, comprising:

[0028] A size acquisition module is configured to acquire size information of the pit, wherein the size information includes depth, length and width, and the size information is determined by using the method for detecting a road surface pit described above;

[0029] A type determination module is configured to determine the type of the pit as an impassable pit type according to the size information and a preset size threshold;

[0030] A distance determination module is configured to determine the distance between the pit and the vehicle according to the point cloud data corresponding to the position information.

[0031] A strategy determination module is configured to determine a corresponding target pre-warning strategy according to the distance.

[0032] The strategy execution module is configured to control the engineering machinery to perform a corresponding early warning action according to the target early warning strategy.

[0033] The sixth aspect of the embodiment of the present application provides a vehicle, comprising: an image acquisition device; a point cloud data acquisition device; and the device for detecting road surface potholes or the device for road surface pothole early warning.

[0034] The seventh aspect of the embodiment of the present application provides a machine readable storage medium, the machine readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the method for detecting road surface potholes or the method for road surface pothole early warning.

[0035] The technical solution described above, by acquiring the color image and the point cloud data corresponding to the to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device respectively, and by the improved YOLOv5 model pre-trained for pothole detection on the color image, the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced with a lightweight convolutional neural network to obtain the position information of the circumscribed rectangular region corresponding to the pothole, and the depth of the pothole is determined according to the point cloud data corresponding to the position information of the circumscribed rectangular region and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, and the length and width of the pothole are determined according to the position information of the circumscribed rectangular region, the technical solution described above can accurately detect the size information such as the depth of the pothole, and the improved YOLOv5 model can be used to detect the pothole of the road surface, so that the model is lightweight, the network parameter quantity and the number of floating point operations are reduced, the model parameter calculation amount and the calculation cost are reduced, and the detection speed and the real-time performance of the detection are improved.

[0036] Other features and advantages of the embodiment of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings are included to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used together with the following specific implementation to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the drawings:

[0038] Figure 1 The flowchart of the method for detecting road surface potholes in the embodiment of the present application is schematically shown;

[0039] Figure 2 The flowchart of the method for road surface pothole early warning in the embodiment of the present application is schematically shown;

[0040] Figure 3A flowchart schematically shows a method for detecting a road pit and a road pit early warning in another embodiment of the present application;

[0041] Figure 4 A structural schematic diagram of a lightweight convolutional neural network in an embodiment of the present application is schematically shown;

[0042] Figure 5 A structural schematic diagram of a lightweight convolutional neural network in another embodiment of the present application is schematically shown;

[0043] Figure 6 A structural schematic diagram of an ECA attention mechanism in an embodiment of the present application is schematically shown;

[0044] Figure 7 A structural schematic diagram of an improved YOLOv5 model in an embodiment of the present application is schematically shown;

[0045] Figure 8 A schematic diagram of a pit and a circumscribed rectangular region in an embodiment of the present application is schematically shown;

[0046] Figure 9 A structural block diagram of a device for detecting a road pit in an embodiment of the present application is schematically shown;

[0047] Figure 10 A structural block diagram of a device for road pit early warning in an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0048] The specific embodiments of the embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the embodiments of the present application, and are not intended to limit the embodiments of the present application.

[0049] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative positional relationship, movement condition, etc. between components in a certain specific posture (as shown in the drawings), and if the specific posture changes, the directional indications will also change accordingly.

[0050] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features indicated or implicitly indicating the number of technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application.

[0051] Figure 1 The flowchart of the method for detecting road potholes in an embodiment of the present application is schematically shown. As shown in the figure, Figure 1 In the embodiments of the present application, a method for detecting road potholes is provided, which is applied to a vehicle, the vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the method is taken as an example to illustrate the processor, which can include the following steps:

[0052] Step S102, the color image and the point cloud data corresponding to the road surface to be detected collected by the image acquisition device and the point cloud data acquisition device respectively are obtained.

[0053] Step S104, the color image is detected by the pre-trained pothole detection model to obtain the position information of the circumscribed rectangular region corresponding to the pothole, wherein the pothole detection model is the improved YOLOv5 model, and the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network.

[0054] Step S106, according to the position information corresponding to the point cloud data and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, the depth of the pothole is determined.

[0055] Step S108, according to the position information corresponding to the point cloud data, the length and width of the pothole are determined.

[0056] It can be understood that the image acquisition device is used for image acquisition of the road surface, and can include a camera and the like. The point cloud data acquisition device is used for acquiring corresponding point cloud data of the road surface, and can include a laser radar and the like. The road surface to be detected is a road surface that needs to be detected for potholes. The circumscribed rectangular region corresponding to the pothole is a region surrounded by a circumscribed rectangle of the pothole, for example, can be a minimum circumscribed rectangular frame surrounding the pothole, and can also be referred to as a two-dimensional boundary rectangular frame. The position information of the circumscribed rectangular region refers to the two-dimensional position information of the circumscribed rectangular region, and can specifically include corner position information and / or center position information and / or rectangular boundary position information of the circumscribed rectangular region. The point cloud data corresponding to the position information is three-dimensional position information, i.e., three-dimensional coordinates, which is the projection of two-dimensional position information in a three-dimensional coordinate system (for example, a laser radar coordinate system). The peripheral region of the circumscribed rectangular region is a nearby region located outside the circumscribed rectangular region, for example, a rectangular region obtained by expanding the circumscribed rectangular region by 1.1 times or 1.2 times belongs to the peripheral region of the circumscribed rectangular region.

[0057] The pre-trained pothole detection model is a model trained in advance for detecting potholes on the road surface in a color image. The pothole detection model is an improved YOLOv5 model. In the improved YOLOv5 model, the C3 feature extraction module in the backbone network is replaced with a lightweight convolutional neural network. Specifically, the lightweight convolutional neural network ShuffleNetV2 can include two feature extraction units, Shufflev2-Block-a unit and Shufflev2-Block-b unit. In the Shufflev2-Block-a unit, the input C-channel feature map is divided into two paths for convolution operation, and then fused by Concat operation after the convolution operation is completed. After channel shuffling, a 2C-channel feature map is output. The network width is expanded with less parameters. In the Shufflev2-Block-b unit, the input C-channel feature map is divided into two parts, C1 and C2. The C1 channel feature map is not changed, and the C2 channel feature map is processed through three convolution layers with the same input and output channels, including two 1X1 ordinary convolution Conv2D and one 3X3 depth separable convolution DWConv. After the two branches are completed, the channels are concatenated by Concat operation, fused into the initial C-channel feature map, and information interaction is performed by channel shuffling.

[0058] Specifically, the processor can acquire a color image corresponding to the to-be-detected road surface and point cloud data corresponding to the to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device at the same time, so that the color image and the point cloud data are time-synchronized and aligned. And input the color image into the pre-trained pit detection model, that is, extract features of the color image through the pit detection model to obtain the position information of the circumscribed rectangular region corresponding to the pit output by the pit detection model on the color image, wherein the pit detection model is an improved YOLOv5 model, and the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network. Further, the processor can determine the length and width of the pit according to the point cloud data corresponding to the position information of the circumscribed rectangular region corresponding to the pit and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, for example, compare the height values in the point cloud data corresponding to the position information of the circumscribed rectangular region corresponding to the pit and the height values in the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, calculate the difference value of the two, and determine the maximum height difference value as the depth of the pit. Thus, the length and width of the pit can be determined according to the point cloud data corresponding to the position information of the circumscribed rectangular region corresponding to the pit, for example, the actual length of the length of the circumscribed rectangular region in three-dimensional coordinates can be determined as the length of the pit, and the actual width of the width of the circumscribed rectangular region in three-dimensional coordinates can be determined as the width of the pit.

[0059] The above method for detecting road surface pits can acquire a color image corresponding to the to-be-detected road surface and point cloud data corresponding to the to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device at the same time, and detect pits through a pre-trained improved YOLOv5 model, the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network, to obtain the position information of the circumscribed rectangular region corresponding to the pit, and determine the depth of the pit according to the point cloud data corresponding to the position information of the circumscribed rectangular region and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, and determine the length and width of the pit according to the position information of the circumscribed rectangular region. The above technical solution can accurately detect the depth and other size information of the pit, and the improved YOLOv5 model can be used to detect the pit of the road surface, which can make the model lightweight, reduce the number of network parameters and the number of floating-point operations, reduce the amount of model parameter calculation and calculation cost, and improve the detection speed and real-time performance of the detection.

[0060] In one embodiment, the lightweight convolutional neural network can include an ECA attention mechanism.

[0061] It is understood that the C3 feature extraction module in the backbone network of the existing YOLOv5 model includes the SE attention mechanism. The lightweight convolutional neural network in the backbone network of the improved YOLOv5 model in this embodiment of the invention adopts the ECA attention mechanism, which can further reduce the computational amount and cost of model parameters while ensuring the detection accuracy of the algorithm, thereby improving the detection speed and real-time performance.

[0062] Specifically, an attention mechanism (Efficient Channel Attention, ECA) is added to the lightweight convolutional neural network ShuffleNetV2 to create a new feature extraction network, ShuffleECA-Net, which is a lightweight convolutional neural network based on the ECA attention mechanism. This includes: improving the ECA attention mechanism by combining it with the Shufflev2-Block-a unit and the Shufflev2-Block-b unit respectively to form the ShuffleECAModule_1 module and the ShuffleECAModule_2 module, and using the ShuffleECAModule_1 module and the ShuffleECAModule_2 module to build the ShuffleECA-Net backbone network.

[0063] Compared to the SE attention mechanism used in the original YOLOv5 model, the ECA attention mechanism avoids frequent dimensionality reduction operations while effectively adjusting feature weights. After global average pooling, the ECA attention mechanism uses a fast 1D convolution with kernel k to capture the interaction information of each channel and its k neighbors. The kernel k is adaptively obtained by being proportional to the channel dimension, and the parameter complexity is further reduced by sharing the weights of all channels. In this calculation, a w function with K*C parameters is used. k Arrays for learning attention, channels y i weight ω i Only y is considered in the calculation process i The interaction information with k neighbors is calculated using the following formula:

[0064]

[0065] in Indicates y i The set of k neighboring channels, where σ is the Sigmoid activation function.

[0066] Weight sharing occurs after information exchange, and the calculation formula is as follows:

[0067]

[0068] The calculation relies on a fast 1D convolution kernel in the ECA structure, and the formula is as follows:

[0069] ω=σ(C1D k (y))

[0070] wherein C1D is a fast 1D convolution, and the cross-channel information interaction attention can effectively improve the efficiency of attention without increasing additional calculation parameters.

[0071] In one embodiment, the path aggregation network in the neck network of the improved YOLOv5 model is replaced by a bi-directional feature pyramid network.

[0072] Understandably, the bi-directional feature pyramid network (BiFPN) is used instead of the path aggregation network (PANet) in the original YOLOv5 model, and the PANet essentially only adds different features, which will lead to the weights of different size features of the same type in the fused output features being unequal. Since the contribution weights of different input features to the output feature map should be different at each node in the feature fusion process, in road pit detection, the shape of the pit is long and discontinuous, and the network needs to have strong feature extraction capability. BiFPN performs top-down deep feature fusion and bottom-up shallow feature fusion through bi-directional connection and weighted feature fusion, skips some intermediate layers, connects and fuses feature layers of different scales, and introduces learning weights to learn the importance of different input features, thereby enhancing the feature extraction capability of the network. BiFPN assigns different weight information to each feature map, proposes a fast normalization fusion weighting method, and uses a ReLU activation function to ensure that each weight is non-negative, and then adds a very small ε (ε = 0.0001) to ensure the stability of the data.

[0073] In one embodiment, the position information includes corner position information and center position information; the color image is detected by the pre-trained pit detection model to obtain the position information of the circumscribed rectangular region corresponding to the pit, including: the color image is detected by the pre-trained pit detection model to obtain the corner position information of the circumscribed rectangular region corresponding to the pit; and the center position information of the circumscribed rectangular region is determined according to the corner position information.

[0074] Specifically, the processor can detect the color image through the pre-trained pit detection model to obtain the corner point position information of the circumscribed rectangular region corresponding to the pit, that is, the position information of the four corner points of the circumscribed rectangular region in the image coordinate system, and calculate the center position information of the center of the circumscribed rectangular region according to the position information of the four corner points. For example, the coordinates of the four corner points of the circumscribed rectangular region are A, B, C and D, respectively, wherein (u1, v1) represents the position of the upper left corner A of the circumscribed rectangular region in the image coordinate system of the color image, (u2, v2) represents the position of the lower right corner D of the circumscribed rectangular region in the image coordinate system of the color image, and the coordinates of the center point E of the circumscribed rectangular region are further calculated

[0075] In one embodiment, according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, the depth of the pit is determined, comprising: obtaining a first height value in the point cloud data corresponding to the center position information and a second height value in the point cloud data corresponding to the peripheral region of the circumscribed rectangular region; determining the absolute value of the difference between the first height value and the second height value to obtain the depth of the pit.

[0076] It can be understood that the point cloud data corresponding to the peripheral region of the circumscribed rectangular region can be the point cloud data corresponding to any one point in the peripheral region of the circumscribed rectangular region. For example, the point cloud data corresponding to the peripheral region of the circumscribed rectangular region can be the point cloud data corresponding to any one point in the rectangular region obtained by expanding the circumscribed rectangular region by 1.1 times. At this time, the second height value is the height value corresponding to the point. In addition, the point cloud data corresponding to the peripheral region of the circumscribed rectangular region can also be the point cloud data corresponding to multiple points in the peripheral region of the circumscribed rectangular region. For example, the point cloud data corresponding to the peripheral region of the circumscribed rectangular region can be the point cloud data corresponding to the four corner points in the rectangular region obtained by expanding the circumscribed rectangular region by 1.1 times. At this time, the second height value is the average of the height values in the point cloud data corresponding to the four corner points. The first height value is the height value of the center position information of the circumscribed rectangular region in the three-dimensional coordinate.

[0077] Specifically, the processor can obtain a first height value in the point cloud data corresponding to the center position information and a second height value in the point cloud data corresponding to the peripheral region of the circumscribed rectangular region, calculate the difference between the first height value and the second height value, and take the absolute value of the difference, so as to obtain the depth of the pit.

[0078] The pit on the road surface seriously affects the driving comfort and safety of the vehicle. The driver of the vehicle has a blind spot in the field of view when driving. If the vehicle encounters a pit when driving or working, it is easy to cause the machine to overturn, lose stability, and lose balance, and even cause a safety accident. Therefore, a way is needed to warn the pit on the road surface in advance.

[0079] Figure 2 A flowchart of a method for road pit hole early warning in an embodiment of the present application is shown schematically. As shown in Figure 2 The embodiment of the present application provides a method for road pit hole early warning, which is applied to a vehicle. Taking a processor as an example, the method can include the following steps:

[0080] In step S202, size information of the pit hole is acquired.

[0081] In step S204, the type of the pit hole is determined as a non-passable pit hole type according to the size information and a preset size threshold.

[0082] In step S206, a distance between the pit hole and the vehicle is determined according to the point cloud data corresponding to the position information.

[0083] In step S208, a corresponding target early warning strategy is determined according to the distance.

[0084] In step S210, the engineering machinery is controlled to perform a corresponding early warning action according to the target early warning strategy.

[0085] It can be understood that the size information can include a depth, a length and a width, and the size information of the pit hole is determined by the method for detecting the road pit hole in the above embodiment. The preset size threshold is a preset size threshold, which can specifically include a preset depth threshold, a preset length threshold and a preset width threshold. The non-passable pit hole type is a pit hole type that the vehicle cannot pass through. The target early warning strategy is a vehicle early warning mode determined based on the size information of the currently acquired road pit hole.

[0086] Specifically, the processor can determine the size information (including the depth, length, and width) of the pit according to the method for detecting the pit in the above-mentioned embodiments, and determine the type of the pit as the impassable pit type according to the size information and the preset size threshold, for example, when the depth in the size information is greater than the preset depth threshold, it can be determined that the type of the pit is the impassable pit type, or when the length in the size information is greater than the preset length threshold, it can be determined that the type of the pit is the impassable pit type, or when the width in the size information is greater than the preset width threshold, it can be determined that the type of the pit is the impassable pit type. After determining that the type of the pit is the impassable pit type, the processor can determine the distance between the pit and the vehicle according to the point cloud data corresponding to the position information of the circumscribed rectangular region of the pit, for example, obtain the projection coordinates (x_lidar, y_lidar, z_lidar) in the laser radar coordinate system in the n*n (n is 3) neighborhood of the boundary coordinates of the circumscribed rectangular region close to the vehicle, and take the average value of y_lidar as the distance between the pit and the vehicle. Further, the processor can select the corresponding target warning strategy according to the distance according to the corresponding relationship between the distance and the warning strategy, and control the engineering machinery to perform the corresponding warning action according to the target warning strategy, for example, when the distance is 1 meter, the corresponding warning strategy is to stop, and when the distance is 2 meters, the corresponding warning strategy is to slow down and issue an alarm sound.

[0087] The above-mentioned method for road pit warning, by obtaining the size information of the pit, and determining the type of the pit as the impassable pit type according to the size information and the preset size threshold, and then determining the distance between the pit and the vehicle according to the point cloud data corresponding to the position information, and determining the corresponding target warning strategy according to the distance, so as to control the engineering machinery to perform the corresponding warning action according to the target warning strategy. The above-mentioned technical solution makes a judgment on the passability of the pit, for the impassable pit type, determines different target warning strategies according to the distance between the vehicle and the pit, and performs warning operation based on the target warning strategy, which can remind the driver in real time about the situation of the road surface blind area in front of the vehicle, and timely remind the driver to pay attention to slow down and avoid, so as to avoid the vehicle from driving into the pit and causing the vehicle to overturn, lose stability, and imbalance, etc., avoid the occurrence of safety accidents, and improve the comfort and safety during the driving of the vehicle.

[0088] In one embodiment, determining the type of the pit as the impassable pit type according to the size information and the preset size threshold comprises: in the case that the depth, length, and width of the pit are all greater than the corresponding preset size threshold, determining the type of the pit as the impassable pit type.

[0089] Specifically, when the processor determines that the depth of the pit is greater than the preset depth threshold, the length of the pit is greater than the preset length threshold, and the width of the pit is greater than the preset width threshold, it can be determined that the type of the pit is the impassable pit type.

[0090] In one embodiment, the corresponding target warning strategy is determined according to the distance, including: determining the distance interval where the distance is located; and determining the warning strategy corresponding to the distance interval as the target warning strategy.

[0091] Specifically, the processor can determine the distance interval where the distance is located, and determine the corresponding warning strategy, i.e., the target warning strategy, according to the determined distance interval based on the pre-determined correspondence between the distance interval and the warning strategy.

[0092] In one embodiment, the distance interval includes a first distance interval, a second distance interval, and a third distance interval from far to near, the warning strategy corresponding to the first distance interval includes issuing prompt information indicating that there is an impassable pit on the road ahead, the warning strategy corresponding to the second distance interval includes issuing prompt information indicating that the vehicle has approached the impassable pit on the road ahead and needs to change the vehicle's driving track, and the warning strategy corresponding to the third distance interval includes controlling the vehicle to slow down or stop.

[0093] It can be understood that the prompt information indicating that there is an impassable pit on the road ahead can be realized by means of indicator light flashing, buzzer sound alarm, and text prompt, etc. The prompt information indicating that the vehicle needs to change the driving track is usually distinguished from the prompt information indicating that there is an impassable pit on the road ahead, and can be realized by means of indicator light flashing, buzzer sound alarm, and text prompt, etc.

[0094] Specifically, three distance intervals can be set according to the distance from far to near, which are the first distance interval, the second distance interval, and the third distance interval. When the distance between the pit and the vehicle is located in the first distance interval, the processor can issue prompt information indicating that there is an impassable pit on the road ahead. When the distance between the pit and the vehicle is located in the second distance interval, the processor can issue prompt information indicating that the vehicle has approached the impassable pit on the road ahead and needs to change the vehicle's driving track. When the distance between the pit and the vehicle is located in the third distance interval, the processor can control the vehicle to slow down or stop.

[0095] In order to obtain the pit condition of the road in front of the aerial work vehicle in real time and effectively, and improve the driving safety of the aerial work vehicle, a specific embodiment of the present application provides a method for detecting a road pit and a road pit warning, which adopts a multi-sensor fusion detection technology of a camera + laser radar, extracts comprehensive features such as the width, length, pit depth of the road pit area and the closest distance between the pit and the vehicle, makes a different level of warning strategy according to the width, length, pit depth of the road pit area and the closest distance between the pit and the vehicle, and performs a warning operation of the corresponding warning level based on the warning strategy.

[0096] The camera has the advantages of low cost, can provide rich color and texture information, and is helpful for target recognition and classification, but it is difficult to obtain accurate three-dimensional information. The laser radar has the advantages of long detection distance, and can provide accurate distance and shape information of the target, and has high stability and good robustness. By fusing the data of the camera and the laser radar, the advantages of the two can be comprehensively utilized, more accurate and robust target recognition can be realized, reliable pit depth and distance information can be obtained, and the robustness and reliability of the perception system are improved.

[0097] Specifically, as shown in Figure 3 The method for detecting a road pit and a road pit warning provided by the specific embodiment of the present application can include the following steps:

[0098] Step 1: install a 2D camera and a 3D laser radar on an aerial work vehicle and perform joint calibration and calibration.

[0099] Step 2: the 2D camera and the 3D laser radar respectively shoot the road in front of the vehicle to obtain an RGB image and 3D point cloud data, and send them to a vehicle-mounted server for data calculation and processing.

[0100] Step 3: pre-process the RGB image and import it into a pre-trained target detection neural network for pit detection.

[0101] Step 4: if no pit is detected, repeat step 1, and if a pit is detected, obtain the two-dimensional boundary rectangular frame (bounding box) position information of the pit in the RGB image according to the target detection neural network.

[0102] Step 5: associate the two-dimensional boundary rectangular frame of the visually detected pit and the pit surrounding road coordinates to the 3D point cloud data in the radar coordinate system.

[0103] Step 6: obtain the position of the pit point cloud and the pit surrounding road point cloud in the radar coordinate system, and obtain the distance between the pit and the vehicle, and obtain the depth of the pit by comparing the height difference of the pit and the surrounding flat road in the radar coordinate system.

[0104] Step seven: according to the two-dimensional boundary rectangular frame position information of the pit, the minimum circumscribed rectangle of the pit is calculated to obtain the width and length of the road pit area.

[0105] Step eight: according to the width, length, pit depth and the closest distance between the pit and the vehicle, a different level of early warning strategy is made.

[0106] Step nine: according to the early warning strategy of step eight, the early warning operation of the corresponding early warning level is executed.

[0107] Specifically, after mapping the two-dimensional rectangular frame of the pit detected on the RGB image and the pit peripheral area to the 3D point cloud data of the laser radar, the 3D point (x, y, z) of the laser radar corresponding to the coordinate (u, v) on the RGB image is obtained.

[0108] In step one, after the high-altitude operation vehicle is installed with the fixed 2D camera and the laser radar, the pixels in the data of the calibration board area collected by the 2D camera and the laser radar are one-to-one corresponding, so as to complete the joint calibration and calibration. The joint calibration and calibration algorithm is as follows: assuming that the RGB image data obtained by the 2D camera is represented by (u, v), and the 3D point obtained by the laser radar is represented by (x, y, z), the relationship between the 3D point (x, y, z) and the 2D point (u, v) is:

[0109]

[0110] Wherein the matrix (f u , f v , u0, v0) is the internal parameter of the 2D camera, which can be obtained according to Zhang Zhengyou calibration method, R is the rotation matrix, and T is the translation vector, which can be obtained by using Autoware and other joint calibration and calibration tools.

[0111] In step two, the laser radar is responsible for collecting 3D point cloud data, and the 2D camera is an RGB camera responsible for collecting RGB image data. The laser radar and the 2D camera collect 3D point cloud data and RGB image data respectively in real time, and send them to the vehicle-mounted server for real-time data calculation and processing. The sampling rates of the 2D camera and the 3D laser radar are different, and the sampling rate of the camera is greater than that of the laser radar. The time stamps of collecting 3D point cloud data and RGB image data are recorded, and the time stamps of the 3D point cloud data are selected as the reference to select the closest RGB image data, so as to realize the time synchronization and alignment of the laser radar data and the camera data. Further, the vehicle-mounted server pre-processes the obtained RGB image, including filtering and noise reduction and changing the size to the input size required by the target detection neural network model.

[0112] The hole detection in step three on the RGB image adopts a hole detection neural network model, which provides a construction method of a lightweight hole target detection neural network model based on YOLOv5 Figure 2 ) including:

[0113] a. Replace the C3 module in the backbone network (Backbone) of YOLOv5 with a lightweight convolutional neural network ShuffleNetV2, which greatly reduces the model parameters and computational cost, and adds an ECA (Efficient Channel Attention) attention mechanism in ShuffleNetV2 to construct a ShuffleNetV2-ECA backbone network.

[0114] b. Change the path aggregation network PANet (Path Aggregation Network) layer in the neck network neck of YOLOv5 to a bidirectional feature pyramid network BiFPN.

[0115] c. Replace the Bounding Box prediction frame regression loss function EIoU in the head network of YOLOv5 with the LEIOU loss function.

[0116] Specifically, the C3 module in the original backbone network of YOLOv5 is replaced with a ShuffleNetV2 module, as shown in Figure 4 , which includes a Shufflev2-Block-a unit and a Shufflev2-Block-b unit; wherein:

[0117] In the Shufflev2-Block-a unit, the input C-channel feature map is divided into two paths for convolution operation, and after the convolution operation is completed, the Concat operation is used for fusion, and the 2C-channel feature map is output after channel shuffling, which expands the network width under the premise of adding fewer parameters.

[0118] In the Shufflev2-Block-b unit, the input C-channel feature map is divided into two parts, C1 and C2, and the C1 channel feature map is not changed, and the C2 channel feature map is passed through three convolution layers with the same input and output channels, including two 1X1 ordinary convolution Conv2D and one 3X3 depth separable convolution DWConv; After the two branches are completed, the Concat operation is performed to concatenate the channels, and the initial C-channel feature map is fused, and information interaction is performed using channel shuffling.

[0119] Further, an ECA attention mechanism is added to the ShuffleNetV2 module to create a new feature extraction network ShuffleECA-Net, including: combining the ECA attention mechanism with the Shufflev2-Block-a unit and the Shufflev2-Block-b unit respectively to improve the ShuffleECAModule_1 module and the ShuffleECAModule_2 module as shown below Figure 5 The ShuffleECA-Net backbone network is constructed by using the ShuffleECAModule_1 module and the ShuffleECAModule_2 module, which can make the model lightweight, reduce the network parameter quantity and the floating point operation times, thereby improving the detection speed, and enable the target detection neural network to be transplanted to the terminal device on the basis of ensuring the detection accuracy of the algorithm.

[0120] The overall structure of the ECA (Efficient Channel Attention) attention mechanism is as follows Figure 6 Compared with the SE attention used by the original YOLOv5, the ECA attention can well adjust the feature weights while avoiding frequent dimension reduction operations. After global average pooling, the ECA attention mechanism uses a fast 1D convolution with a convolution kernel of k to obtain the interaction information between each channel and its k neighbors, where the convolution kernel k is proportional to the channel dimension and is obtained from adaptation, and the weight is shared by all channels to further reduce the parameter complexity. In this calculation process, a w k array containing K*C parameters is used to learn the attention, and the weight ω i of the channel y i is calculated. i Only the interaction information between y i and its k neighbors is considered in the calculation process, and the calculation formula is as follows:

[0121]

[0122] where represents the set of k adjacent channels of y i , and σ is the Sigmoid activation function.

[0123] After information interaction, weight sharing is performed, and the calculation formula is as follows:

[0124]

[0125] In the calculation process, the fast 1D convolution with a convolution kernel of k (the size of k is adaptive) in the ECA structure is relied on, and its formula is as follows:

[0126] ω=σ(C1D k (y))

[0127] wherein C1D is a fast 1D convolution, and this cross-channel information interaction attention can effectively improve the efficiency of attention without increasing additional calculation parameters.

[0128] Further, a bidirectional feature pyramid network BiFPN is used instead of the original YOLOv5 PANet, and the PANet essentially only adds different features, which will cause the weights of different size features of the same type in the fused output features to be unequal. Since in the feature fusion process, the contribution weights of different input features to the output feature map should be different at each node, in the road pit detection, the shape of the pit is long and discontinuous, and the network needs to have strong feature extraction capability. BiFPN performs top-down deep feature and bottom-up shallow feature fusion through bidirectional connection and weighted feature fusion, skips some intermediate layers, connects and fuses feature layers of different scales, and introduces learning weights to learn the importance of different input features, thereby enhancing the feature extraction capability of the network. BiFPN gives different weight information to each feature map, proposes a fast normalization fusion weighted method, and uses a ReLU activation function to ensure that each weight is non-negative, and then adds a very small ε (ε = 0.0001) to ensure the stability of the data, as shown in the following formula:

[0129]

[0130] Finally, in the model training stage, the loss function and the prediction box screening method are optimized. The YOLOv5 model mainly uses the CIOU loss, which considers the distance between the center point of the prediction box and the center line of the real boundary box and the aspect ratio of the prediction box and the real box. Since the types of road pits have diversity, it is difficult to accurately predict the real boundary box by using the CIOU loss. In the method of the patent, the EIoU loss function is used as the prediction box regression loss function CIOU of the improved YOLOv5 algorithm. The EIOU loss refers to the calculation method of the overlap loss and the center distance loss in the CIOU, but the width and height loss uses the minimum value of the difference between the width and height of the prediction box and the real boundary box, which makes the model converge faster and obtain higher precision.

[0131]

[0132] wherein IoU is the intersection over union of the predicted boundary box and the real box, ρ 2 (b,b gt is the distance between the center point of the predicted box and the center point of the ground truth boundary box. c is the diagonal length of the minimum enclosing box covering the predicted box and the ground truth boundary box, C w and C his the width and height of the minimum bounding box covering the prediction box and the ground truth bounding box. However, there is a problem of data sample imbalance in the training data set of the road pit, which will make the number of high-quality samples with smaller regression error in the image far less than that of low-quality samples with larger error. Poor quality samples will produce larger gradients and affect the training process. Therefore, Focal Loss is introduced to solve the problem of sample imbalance in regression, and the precision loss L Focal-EIOU is improved using the focal loss L Focal-EIOU = IOUγL EIOU , where γ is a focus parameter representing the degree of suppression of outliers.

[0133] The RGB image in step three is subjected to pit detection by a pit detection model, which is an improved YOLOv5 model, as shown in Figure 7 , including mainly backbone, neck, and head. The RGB image is input into the improved YOLOv5 model for training and detection, and the main process can be as follows:

[0134] 1) The above RGB image is input into the backbone network, which is processed as follows:

[0135] The above RGB image is sequentially subjected to Focus slicing operation, CBL operation, CSP1_1 operation, CBL operation, CSP1_3 operation, ShuffleECAModule_2 and ShuffleECAModule_1 operation, and the first layer feature map of the input image is output; the first layer feature map of the input image is sequentially subjected to ShuffleECAModule_2 and ShuffleECAModule_1 operation, and the second layer feature map of the input image is output; the second layer feature map of the input image is sequentially subjected to ShuffleECAModule_2 and ShuffleECAModule_1 operation, and the third layer feature map of the input image is output.

[0136] 2) In the above neck network (Neck):

[0137] The third layer feature map of the input image is input into the Upsample up-sampling operation and the second layer feature map of the input image is concatenated, and then the fourth layer feature map of the input image is output after sequentially passing through C3_3 and CBL operations; the fourth layer feature map is subjected to Upsample up-sampling and the first layer feature map of the input image is concatenated, and the fifth layer feature map is obtained after C3_3 operation.

[0138] The fifth layer feature map is subjected to a CBL operation and is concatenated with the fourth layer feature map described above, and after a C3_3 operation, a sixth layer feature map of the input image is obtained.

[0139] The sixth layer feature map is subjected to a CBL operation and is concatenated with the third layer feature map described above, and after a C3_3 operation, a seventh layer feature map of the input image is obtained.

[0140] The fifth layer feature map, the sixth layer feature map and the seventh layer feature map described above are subjected to a Conv2d convolution operation respectively, and are subjected to a Detect processing, to obtain an eighth layer feature map of the image, at this time, the detection of the target in the input image and the framing of the target object have been completed.

[0141] 3) In the above head network (Head):

[0142] The eighth layer feature map is subjected to a loss function calculation of the target Boundingbox, and L Focal-EIOU The NMS non-maximum suppression operation is performed in a manner, and an output size is 3 feature maps of different sizes, a depth of the feature map is 3*(N+1+4), (N is a number of categories in a data set, 1 represents a confidence value, and 4 represents a center point coordinate and its width and height), and three different feature layers output by an output end give a regression bounding box and a confidence; a method of non-maximum suppression is used to filter out repeated and higher than a set threshold bounding box, to obtain a prediction box, the prediction box is compared with a labeled box, and L Focal-EIOU The bounding box loss is calculated, and according to the loss function, back propagation is performed, and the weight of the improved YOLOv5 model in the training process is adjusted.

[0143] The embodiment of the application also provides a construction method of a lightweight pit target detection neural network based on YOLOv5, including the following training steps:

[0144] Step S1, video data sets of road pit in different scenes and different time periods are collected by a camera fixed on an aerial work vehicle.

[0145] Step S2, the video is converted into frame pictures, and data enhancement preprocessing is performed by using Mosaic; the enhanced data set is divided into a training set and a test set in a certain proportion, after the data set is divided, the pictures are manually labeled by using Labelimg software, the positions of the pits are labeled by using a rectangular frame, and an XML format file is generated.

[0146] Step S3, the training set and the test set are used to construct the lightweight pit target detection neural network model based on YOLOv5 described above, and the pit target detection neural network can extract features of characteristic information on the image through a convolution network.

[0147] Step S4, input the image data to be detected into the optimized YOLOv5 pit hole target detection model to obtain the detection result.

[0148] Specifically, the depth of the pit can include: using the above-mentioned pit detection neural network model to detect the pit in the RGB image, if the pit is detected on the road surface, the two-dimensional boundary rectangular frame position information of the pit in the RGB image is returned, which is represented as (u1, v1, u2, v2), the four coordinates of the rectangular frame are represented as A, B, C, D, as shown in the figure. Figure 8 (u1, v1) represents the position of the predicted rectangular frame upper left corner A in the RGB image coordinate system, (u2, v2) represents the position of the predicted rectangular frame right lower corner D in the RGB image coordinate system, and the center point E coordinate of the pit rectangular frame is further calculated The projection coordinates (x_lidar, y_lidar, z_lidar) in the laser radar coordinate system within the n*n (n can be 3, 5, 7) neighborhood of the center point coordinate (u_mid, v_mid) of the pit rectangular frame detected in the RGB image are associated, and the average value of z_lidar is taken as the height of the pit in the laser radar coordinate system; the pit rectangular frame is enlarged by 1.1 times to obtain the rectangular frame of the envelope pit and the surrounding road surface, and the surrounding road surface of the pit is considered to be flat, and the four coordinates of A', B', C', D' in the laser radar coordinate system are calculated and averaged to obtain the height of the surrounding road surface of the pit in the laser radar coordinate system; further, the height between the pit and the surrounding road surface is calculated, that is, the depth H of the pit.

[0149] The nearest distance between the pit and the vehicle is calculated, the projection coordinates (x_lidar, y_lidar, z_lidar) in the laser radar coordinate system within the n*n (n takes 3) neighborhood of the boundary coordinate (u1

[0150] Calculate the minimum circumscribed rectangle of the pit: according to the association of the pit two-dimensional boundary rectangular frame to the 3D point cloud data set in the radar coordinate system, project the coordinates in the laser radar coordinate system to the ground plane. Assuming that the ground plane is the X-Y plane, z_lidar can be set to 0 to obtain the projection coordinates (x_proj, y_proj, 0) of the pit rectangular frame in the laser radar coordinate system. The projection coordinates (x_proj, y_proj, 0) are regarded as a point cloud data set. RANSAC is used to calculate the minimum circumscribed rectangle, and the detailed steps are as follows:

[0151] Initialize parameters for RANSAC: set the number of iterations and the sample size, define the distance threshold for determining if a point belongs to the minimum enclosing rectangle.

[0152] Iteration process:

[0153] a. Randomly select sample_size points as a sample set for iteration.

[0154] b. Calculate the centroid coordinates (x_center, y_center, z_center) of the sample set.

[0155] c. For each point, calculate its polar coordinate representation relative to the centroid:

[0156] Calculate the distance of the point to the centroid: ρ_i = sqrt((x_i - x_center)^2 + (y_i - y_center)^2)

[0157] Calculate the angle of the point: θ_i = atan2(y_i - y_center, x_i - x_center)

[0158] d. In the polar coordinate representation, use RANSAC to estimate the rotation angle:

[0159] Randomly select two points and calculate their angle difference: θ_diff = θ_2 - θ_1

[0160] Repeat this process multiple times and select the two points with the largest angle difference as the optimal sample

[0161] Calculate the average angle of the optimal sample as the rotation angle: θ_rot = (θ_1 + θ_2) / 2

[0162] e. According to the estimated rotation angle, perform a rotation transformation on the polar coordinates of each point:

[0163] Rotated angle: θ_i' = θ_i - θ_rot

[0164] f. In the rotated polar coordinate set, select the points that belong to the minimum enclosing rectangle according to the distance threshold:

[0165] Find the minimum distance in the polar coordinate set: ρ_min' = min(ρ_i')

[0166] Find the maximum distance in the polar coordinate set: ρ_max' = max(ρ_i')

[0167] g. Convert the polar coordinates of the minimum circumscribed rectangle (p_min', p_max') back to the Cartesian coordinate system to obtain the corner coordinates (x_min', y_min') and (x_max', y_max') of the rectangular frame:

[0168] x_min' = p_min' * cos(0_rot)

[0169] y_min' = p_min' * sin(0_rot)

[0170] x_max' = p_max' * cos(0_rot)

[0171] y_max' = p_max' * sin(0_rot)

[0172] h. Repeat the iteration process several times and select the result with the smallest circumscribed rectangle area as the final hole circumscribed rectangle. Finally, the corner coordinates (x_min, y_min) and (x_max, y_max) of the hole circumscribed rectangle in the laser radar coordinate system and the rotation angle 0_rot are obtained.

[0173] According to the minimum circumscribed rectangle of the hole, the width W = x_max - x_min of the road hole and the length L = y_max - y_min of the road hole are obtained.

[0174] The depth of the hole, the width of the hole, the length of the hole and the set threshold value are compared to determine the passability of the hole: when H > H_limit, L > L_limit or W > W_limit, it is judged as an impassable hole type, and other cases are considered as a passable hole type. For the impassable hole type, determine the warning strategy; the warning strategy is divided into different warning levels according to the distance between the vehicle and the hole, prompting the driver of the urgency; the warning levels include first-level warning, second-level warning and third-level warning, such as detecting an impassable hole in front of the road, if the distance between the aerial work vehicle and the hole is between 1.5-2.5 meters, set it as a first-level warning; if the distance between the hole and the aerial work vehicle is between 0.5-1.5 meters, set it as a second-level warning, and if the distance between the hole and the aerial work vehicle is less than 0.5 meters, set it as a third-level warning.

[0175] The early warning operation corresponding to the early warning level is performed based on the early warning strategy, and different early warning operations correspond to different early warning levels. For example, the first-level early warning only indicates light flickering for prompting the driver that there is a non-passable pit in the road surface in front of the vehicle, when the second-level early warning is reached, the corresponding light flickering and buzzer alarm are used for prompting the driver that the vehicle has approached the non-passable pit in the road surface in front of the vehicle, and the driving track of the vehicle needs to be changed in time; in addition to light flickering and buzzer alarm, the third-level early warning operation also includes active intervention control of vehicle deceleration or even stop.

[0176] The technical scheme provided by the embodiment of the present application adopts a multi-sensor fusion detection technology of camera + laser radar, extracts comprehensive features such as the pit region width, length, pit depth and the closest distance between the pit and the vehicle, makes different levels of early warning strategies based on the pit region width, length, pit depth and the closest distance between the pit and the vehicle, and performs early warning operations corresponding to the early warning level based on the early warning strategy. Specifically, the present application adopts an improved YOLOv5 convolutional neural network, which reduces the calculation amount of network parameters and improves the real-time performance of detection. In addition, the technical scheme can detect pits on the road surface in the driving area in front of the aerial work vehicle in real time, make a judgment on the passability of the pit, make different levels of early warning strategies for the non-passable pit type in combination with the distance between the vehicle and the pit, perform early warning operations based on the early warning strategy, remind the driver in real time about the situation of the visual blind area of the road surface in front of the vehicle, and timely remind the driver to pay attention to deceleration and avoidance, so as to avoid the vehicle driving into the pit and causing the vehicle to overturn, lose stability, and imbalance, and avoid the occurrence of safety accidents, and improve the comfort and safety of the aerial work vehicle during driving.

[0177] The embodiment of the present application provides a processor configured to execute the method for detecting a road pit according to the above-mentioned embodiments.

[0178] The embodiment of the present application provides a processor configured to execute the method for road pit early warning according to the above-mentioned embodiments.

[0179] As Figure 9 shown, the embodiment of the present application provides a device 900 for detecting a road pit, applied to a vehicle, the vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the device comprises:

[0180] The data acquisition module 910 is configured to acquire the color image and the point cloud data corresponding to the to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device respectively.

[0181] The pit detection module 920 is configured to detect the color image by using the pre-trained pit detection model to obtain position information of an external rectangular region corresponding to the pit; the pit detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network.

[0182] The size determination module 930 is configured to determine a depth of the pit according to point cloud data corresponding to the position information and point cloud data corresponding to a peripheral region of the external rectangular region, and determine a length and a width of the pit according to the point cloud data corresponding to the position information.

[0183] The device for detecting a road surface pit described above can accurately detect the depth and other size information of the pit, and the improved YOLOv5 model can make the model lightweight, reduce the number of network parameters and the number of floating-point operations, reduce the amount of model parameter calculation and the calculation cost, and improve the detection speed and real-time performance of the detection.

[0184] In one embodiment, the lightweight convolutional neural network includes an ECA attention mechanism.

[0185] In one embodiment, a path aggregation network in a neck network of the improved YOLOv5 model is replaced by a bidirectional feature pyramid network.

[0186] In one embodiment, the pit detection module 920 is further configured to detect the color image by using the pre-trained pit detection model to obtain corner position information of the external rectangular region corresponding to the pit, and determine center position information of the external rectangular region according to the corner position information.

[0187] In one embodiment, the pit detection module 920 is further configured to obtain a first height value in point cloud data corresponding to the center position information and a second height value in point cloud data corresponding to a peripheral region of the external rectangular region, and determine an absolute value of a difference between the first height value and the second height value to obtain the depth of the pit.

[0188] As Figure 10As shown, the embodiment of the present application provides a device 1000 for road pit hole early warning, applied to a vehicle, comprising:

[0189] A size acquisition module 1010 is configured to acquire size information of the pit hole, wherein the size information comprises depth, length and width, and the size information is determined according to the method for detecting the road pit hole in the above-mentioned embodiment.

[0190] A type determination module 1020 is configured to determine the type of the pit hole as the impassable pit hole type according to the size information and a preset size threshold.

[0191] A distance determination module 1030 is configured to determine the distance between the pit hole and the vehicle according to the point cloud data corresponding to the position information.

[0192] A strategy determination module 1040 is configured to determine a target early warning strategy according to the distance.

[0193] A strategy execution module 1050 is configured to control the engineering machinery to perform a corresponding early warning action according to the target early warning strategy.

[0194] The device for road pit hole early warning determines the type of the pit hole as the impassable pit hole type according to the size information and a preset size threshold, and then determines the distance between the pit hole and the vehicle according to the point cloud data corresponding to the position information, and determines a target early warning strategy according to the distance, so as to control the engineering machinery to perform a corresponding early warning action according to the target early warning strategy. The above technical solution makes a judgment on the passability of the pit hole, determines different target early warning strategies for the impassable pit hole type in combination with the distance between the vehicle and the pit hole, performs early warning operations based on the target early warning strategy, timely reminds the driver of the situation of the road visual blind area in front of the vehicle, and timely reminds the driver to pay attention to deceleration and avoidance, so as to avoid the vehicle from entering the pit hole and causing the vehicle to overturn, lose stability, and lose balance, and to avoid the occurrence of safety accidents, thereby improving the comfort and safety of the vehicle during driving.

[0195] In one embodiment, the type determination module 1020 is further configured to determine the type of the pit hole as the impassable pit hole type when the depth, length and width of the pit hole are all greater than the corresponding preset size threshold.

[0196] In one embodiment, the strategy determination module 1040 is further configured to determine a distance interval in which the distance is located, and determine the early warning strategy corresponding to the distance interval as the target early warning strategy.

[0197] In one embodiment, the distance interval includes a first distance interval, a second distance interval and a third distance interval from far to near, the pre-warning strategy corresponding to the first distance interval includes issuing prompt information indicating that there is an impassable pit on the front road surface, the pre-warning strategy corresponding to the second distance interval includes issuing prompt information indicating that the vehicle has approached the impassable pit on the front road surface and needs to change the driving track of the vehicle, and the pre-warning strategy corresponding to the third distance interval includes controlling the vehicle to slow down or stop.

[0198] The embodiment of the present application provides a vehicle, which comprises: an image acquisition device; a point cloud data acquisition device; and the device for detecting a road pit in the above-mentioned embodiments.

[0199] The embodiment of the present application provides a vehicle, which comprises: an image acquisition device; a point cloud data acquisition device; and the device for road pit pre-warning in the above-mentioned embodiments.

[0200] Understandably, the vehicle can include but is not limited to an aerial work vehicle and the like engineering vehicle, the image acquisition device can include but is not limited to a 2D camera, and the point cloud data acquisition device can include but is not limited to a laser radar. Further, the vehicle can further include an indicator light and a buzzer and a vehicle-mounted server. In addition, the vehicle can include a power supply, an aerial work vehicle control unit and a road pit detection and pre-warning system of the aerial work vehicle, the power supply can be a 12V / 24V direct current low-voltage power supply, and the power supply can supply power to the road pit detection and pre-warning system of the aerial work vehicle, so that the system can execute the above-mentioned method for detecting a road pit and / or the method for road pit pre-warning.

[0201] In other embodiments, the computing unit of the road pit detection and pre-warning system of the aerial work vehicle is connected with the controller of the aerial work vehicle through CAN communication, and can send a signal for slowing down or stopping to the controller of the aerial work vehicle to assist in controlling the vehicle to slow down or brake.

[0202] The laser radar module is used to acquire point cloud data of the front road surface of the vehicle, and the 2D camera is used to acquire RGB image data of the front road surface of the vehicle. The indicator light and the buzzer are used to perform a pre-warning operation, and the indicator light can flash at different frequencies and the buzzer can emit different frequencies of alarm sound to prompt the driver of a pre-warning level. The laser radar module, the camera module, the indicator light and the buzzer are connected with the vehicle-mounted server, and the vehicle-mounted server comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a road pit detection and pre-warning method of an aerial work vehicle.

[0203] The laser radar is used for collecting point cloud data of the front road surface of the vehicle, the laser radar is directly powered by the vehicle, the collected point cloud data is sent to the vehicle-mounted server for further processing, the laser radar is installed on the driving direction of the aerial work vehicle, and the distance from the ground is about 30cm, and the installation angle is adjusted so that the laser radar can collect point cloud data of the road surface region of 0.3m to 5m in the driving direction of the vehicle.

[0204] The 2D camera is used for collecting RGB image data of the front road surface of the vehicle, the camera is directly powered by the vehicle, the collected RGB image data is sent to the vehicle-mounted server for further processing, the camera is installed on the driving direction of the aerial work vehicle, and the distance from the ground is about 50cm, and the installation angle is adjusted so that the camera can collect image data of the road surface region of 0.3m to 5m in the driving direction of the vehicle.

[0205] The indicator light and the buzzer can receive the control signal of the vehicle-mounted server, the aerial work vehicle executes the indicator light flickering and the sound warning according to the warning strategy when warning, and timely reminds the driver to pay attention to deceleration and avoidance, and improves the comfort and safety of the aerial work vehicle driving. The indicator light and the buzzer can be installed in the working bar of the aerial work vehicle, and the position is convenient for the driver to see or hear. The driver can understand the existence of the pit in the road surface in front of the vehicle and the distance between the vehicle and the pit in time through the prompt of the indicator light and the buzzer, so as to avoid the aerial work vehicle driving into the pit due to the visual blind area, thereby avoiding unnecessary safety accidents.

[0206] The embodiment of the application provides a machine readable storage medium, and the machine readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the method for detecting the road pit in the above embodiment.

[0207] The embodiment of the application provides a machine readable storage medium, and the machine readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the method for detecting the road pit in the above embodiment.

[0208] Those skilled in the art should understand that the embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0209] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0210] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0211] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof. Figure 1 one or more flowcharts and / or blocks in the flowcharts and / or combination thereof.

[0212] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0213] The memory can include non-persistent memory and / or volatile memory, such as a random access memory (RAM) including a cache area for the temporary storage of data. The memory can also include non-volatile memory, such as read only memory (ROM) for storing structural information and / or instruction code to boot an operating system. Access and / or storage of information on the memory can be performed by one or more memory controllers. The memory can also include other volatile memory and / or non-volatile memory, such as flash memory, magnetic computer storage media, optical computer storage media, tape, and / or soft or hard disk drives.

[0214] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0215] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0216] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for detecting a road surface pit, applied to a vehicle, characterized by, The vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the method comprises: acquiring a color image and point cloud data corresponding to a to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device respectively at the same time; detecting the color image by using a pre-trained pit detection model to obtain position information of an external rectangular region corresponding to a pit, wherein the pit detection model is an improved YOLOv5 model, a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network, the lightweight convolutional neural network comprises an ECA attention mechanism, and the position information comprises corner position information and center position information; determining the depth of the pit according to point cloud data corresponding to the position information and point cloud data corresponding to a peripheral region of the external rectangular region; determining the length and width of the pit according to the point cloud data corresponding to the position information; wherein the detection of the color image by using the pre-trained pit detection model to obtain the position information of the external rectangular region corresponding to the pit comprises detection of the color image by using the pre-trained pit detection model to obtain the corner position information of the external rectangular region corresponding to the pit; and the center position information of the external rectangular region is determined according to the corner position information. the determination of the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral region of the external rectangular region comprises: acquiring a first height value in the point cloud data corresponding to the center position information and a second height value in the point cloud data corresponding to the peripheral region of the external rectangular region; and determining an absolute value of a difference between the first height value and the second height value to obtain the depth of the pit.

2. The method of claim 1, wherein, The path aggregation network in the neck network of the improved YOLOv5 model is replaced by a bidirectional feature pyramid network.

3. A method for road hole early warning, applied to a vehicle, characterized in that, The method comprises: acquiring size information of a pit, wherein the size information comprises a depth, a length and a width, and the size information is determined by using the method for detecting a pit on a road surface according to any one of claims 1 to 2; determining that the type of the pit is a non-passable pit type according to the size information and a preset size threshold; determining a distance between the pit and the vehicle according to the point cloud data corresponding to the position information; determining a corresponding target warning strategy according to the distance; controlling the vehicle to perform a corresponding warning action according to the target warning strategy.

4. The method of claim 3, wherein, The determination that the type of the pit is the non-passable pit type according to the size information and the preset size threshold comprises: in a case where the depth, the length and the width of the pit are all greater than a corresponding preset size threshold, determining that the type of the pit is the non-passable pit type.

5. The method of claim 3, wherein, The determination of the corresponding target warning strategy according to the distance comprises: determining a distance interval in which the distance is located; determining that a warning strategy corresponding to the distance interval is the target warning strategy.

6. The method of claim 5, wherein, The distance interval includes a first distance interval, a second distance interval and a third distance interval from far to near, the first distance interval corresponds to a pre-warning strategy including issuing prompt information indicating that there is an impassable pit in the front road surface, the second distance interval corresponds to a pre-warning strategy including issuing prompt information indicating that the vehicle has approached the impassable pit in the front road surface and needs to change the vehicle driving track, and the third distance interval corresponds to a pre-warning strategy including controlling the vehicle to slow down or stop.

7. A processor, comprising: The vehicle is configured to perform the method for detecting a road pit according to any one of claims 1-2 or the method for road pit pre-warning according to any one of claims 3-6.

8. A device for detecting a road surface pit, applied to a vehicle, characterized by, The vehicle is provided with an image acquisition device and a point cloud data acquisition device, comprising: A data acquisition module is configured to acquire a color image and point cloud data corresponding to a to-be-detected road surface collected by the image acquisition device and the point cloud data acquisition device, respectively; A pit detection module is configured to detect the color image by using a pre-trained pit detection model to obtain position information of a circumscribed rectangular region corresponding to a pit; wherein the pit detection model is an improved YOLOv5 model, a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network, the lightweight convolutional neural network includes an ECA attention mechanism, and the position information includes corner point position information and center position information; A size determination module is configured to determine a depth of the pit according to point cloud data corresponding to the position information and point cloud data corresponding to a peripheral region of the circumscribed rectangular region, and determine a length and a width of the pit according to the point cloud data corresponding to the position information; The pit detection module is further configured to: detect the color image by using the pre-trained pit detection model to obtain corner point position information of a circumscribed rectangular region corresponding to a pit; and determine center position information of the circumscribed rectangular region according to the corner point position information; The size determination module is further configured to: acquire a first height value in point cloud data corresponding to the center position information and a second height value in point cloud data corresponding to a peripheral region of the circumscribed rectangular region; and determine an absolute value of a difference between the first height value and the second height value to obtain the depth of the pit.

9. A device for road hole early warning, applied to a vehicle, characterized in that, Comprising: A size acquisition module is configured to acquire size information of a pit, wherein the size information includes a depth, a length and a width, and the size information is determined by the method for detecting a road pit according to any one of claims 1-2; A type determination module is configured to determine a type of the pit as an impassable pit type according to the size information and a preset size threshold; A distance determination module is configured to determine a distance between the pit and the vehicle according to point cloud data corresponding to the position information; A strategy determination module is configured to determine a target pre-warning strategy according to the distance; A strategy execution module is configured to control the vehicle to perform a corresponding pre-warning action according to the target pre-warning strategy.

10. A vehicle characterized by comprising: Comprising: An image acquisition device; A point cloud data acquisition device; And The device for detecting road potholes according to claim 8 or the device for road pothole warning according to claim 9.

11. A machine-readable storage medium having stored thereon a program or instmctions, characterized in that, The program or the instructions are executed by the processor to implement the method for detecting road potholes according to any one of claims 1 to 2 or the method for road pothole warning according to any one of claims 3 to 6.

Images