Ice crack detection method and device based on camera and laser radar

Abstract

The embodiment of the invention provides an ice crack detection method and device based on a camera and a laser radar. The method comprises the following steps: carrying out joint calibration on the camera and the laser radar of a driving platform; acquiring an image of a surrounding ice and snow environment by using the camera after joint calibration; performing visible ice seam detection according to the image to obtain a visible ice seam detection result; acquiring point cloud data of a surrounding ice and snow environment by using the laser radar after joint calibration; performing invisible ice seam detection according to the point cloud data to obtain an invisible ice seam detection result; and converting the visible ice seam detection result and the invisible ice seam detection result to a driving platform coordinate system. In this way, the visible ice cracks and the invisible ice cracks in the ice and snow environment where the driving platform is located can be accurately and efficiently detected by combining the advantages of the camera and the laser radar, and navigation of the driving platform in the ice and snow environment is optimized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an ice crack detection method and device based on a camera and a laser radar. Background Art

[0002] Ice crack detection is an important task in autonomous driving in ice and snow environments, and it is related to the safe driving of intelligent vehicles. However, there are many challenges in ice and snow environments. In particular, ice and snow environments have characteristics such as changeable lighting conditions, complex ground structures, extreme low temperatures, and strong winds. When performing inspection tasks, detection vehicles in ice and snow environments, whether manned or unmanned, will encounter ice cracks, ice pits, ice ridges and other impassable obstacles (hereinafter referred to as ice cracks) during their driving. Therefore, ice cracks must be detected before the vehicle's driving route can be further planned to ensure personnel safety and improve vehicle operation efficiency. Ice crack detection is a cross-integration of multiple fields. At present, there is little attention to the issue of ice crack detection at home and abroad, and no team has conducted research specifically on this topic. Summary of the invention

[0003] In a first aspect, an embodiment of the present invention provides an ice crack detection method based on a camera and a laser radar, the method comprising:

[0004] Jointly calibrate the camera and lidar of the driving platform;

[0005] Use the jointly calibrated camera to obtain images of the surrounding ice and snow environment;

[0006] Perform visible ice crack detection according to the image to obtain visible ice crack detection results;

[0007] Use the jointly calibrated LiDAR to obtain point cloud data of the surrounding ice and snow environment;

[0008] Invisible ice crack detection is performed according to the point cloud data to obtain invisible ice crack detection results;

[0009] The visible ice crack detection results and the invisible ice crack detection results are converted to the driving platform coordinate system.

[0010] In some implementations of the first aspect, jointly calibrating a camera and a laser radar of a driving platform includes:

[0011] The Autoware tool is used to jointly calibrate the camera and lidar of the driving platform.

[0012] In some possible implementations of the first aspect, performing visible ice crack detection according to an image to obtain a visible ice crack detection result includes:

[0013] The image is input into a visible ice crack detection model, and the visible ice crack detection model detects the image to obtain a visible ice crack detection result;

[0014] Among them, the visible ice crack detection model is obtained through the following steps:

[0015] The open source model of pavement crack detection on the Roboflow platform was transferred to learn multiple images of ice and snow environments with visible ice cracks marked, and a visible ice crack detection model was obtained.

[0016] In some possible implementations of the first aspect, performing invisible ice crack detection according to point cloud data to obtain invisible ice crack detection results includes:

[0017] Overlay multiple frames of point cloud data within a preset time length;

[0018] Extract the largest connected area in the superimposed point cloud data;

[0019] Extract point cloud data with vertical height lower than or equal to the vertical height threshold in the maximum connected area;

[0020] Divide the extracted point cloud data into a plurality of annular sectors of equal width and equal angle;

[0021] For any annular sector, the reflectivity average value of the point cloud data in the annular sector is calculated, and a reflectivity threshold is set based on the average value, and the point cloud data in the annular sector whose reflectivity is greater than or equal to the reflectivity threshold is determined as reflectivity abnormal point cloud data;

[0022] The reflectivity anomaly point cloud data of each annular sector are summarized, and the summarized reflectivity anomaly point cloud data are clustered, and each cluster obtained by clustering is determined as an invisible ice crack to obtain an invisible ice crack detection result.

[0023] In some implementations of the first aspect, the method further includes:

[0024] For any annular sector, a two-dimensional plane is fitted according to the point cloud data in the annular sector to obtain a fitted two-dimensional plane, and the height of the point cloud data in the annular sector to the two-dimensional plane is counted, and the height variance is calculated based on this. If the height variance is greater than or equal to the height variance threshold, the annular sector is determined as a non-drivable area;

[0025] Summarize all non-drivable areas and obtain the non-drivable area detection results;

[0026] The visible ice crack detection results and the invisible ice crack detection results are converted to the driving platform coordinate system, including:

[0027] The visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results are converted to the driving platform coordinate system.

[0028] In some possible implementations of the first aspect, converting the visible ice crack detection result, the invisible ice crack detection result, and the non-drivable area detection result to a driving platform coordinate system includes:

[0029] According to the intrinsic parameter matrix of the camera obtained by joint calibration, the visible ice crack detection results in the pixel coordinate system are converted to the camera coordinate system;

[0030] According to the external parameter matrix between the camera and the lidar obtained by joint calibration, the visible ice crack detection results in the camera coordinate system are converted to the lidar coordinate system;

[0031] The visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results in the lidar coordinate system are converted to the driving platform coordinate system.

[0032] In some implementations of the first aspect, the method further includes:

[0033] The visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results in the driving platform coordinate system are visualized.

[0034] In a second aspect, an embodiment of the present invention provides an ice crack detection device based on a camera and a laser radar, the device comprising:

[0035] The calibration module is used to jointly calibrate the camera and lidar of the driving platform;

[0036] An acquisition module is used to acquire images of the surrounding ice and snow environment using the jointly calibrated camera;

[0037] A detection module, used to detect visible ice cracks according to the image and obtain visible ice crack detection results;

[0038] An acquisition module is used to acquire point cloud data of the surrounding ice and snow environment using the jointly calibrated lidar;

[0039] A detection module, used for performing invisible ice crack detection according to point cloud data to obtain invisible ice crack detection results;

[0040] The conversion module is used to convert the visible ice crack detection results and the invisible ice crack detection results into the driving platform coordinate system.

[0041] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0042] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described above.

[0043] In an embodiment of the present invention, the advantages of cameras and lidars can be combined to accurately and efficiently detect visible and invisible ice cracks in the ice and snow environment in which the driving platform is located, thereby optimizing the navigation of the driving platform in the ice and snow environment.

[0044] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0046] Figure 1 A flowchart of an ice crack detection method based on a camera and a laser radar provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of various training curves in the process of transfer learning;

[0048] Figure 3 The semantic segmentation results of visible ice cracks (top: dataset; bottom: real scene);

[0049] Figure 4 It is a schematic diagram of annular sectors;

[0050] Figure 5 The real scene of invisible ice cracks built for the laboratory and the schematic diagram of the invisible ice crack detection results;

[0051] Figure 6 It is a schematic diagram of the operating environment of the ice crack detection method;

[0052] Figure 7 It is a schematic diagram of the detection results of visible ice cracks, invisible ice cracks and non-drivable areas, the detection results of high point areas and the detection results of low point areas;

[0053] Figure 8 A structural diagram of an ice crack detection device based on a camera and a laser radar provided in an embodiment of the present invention;

[0054] Fig. 9 The figure is a structural diagram of an exemplary electronic device capable of implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0057] In order to solve the technical problems arising from the background technology, the embodiments of the present invention provide a method, device, equipment and storage medium for ice crack detection based on camera and laser radar. Specifically, the camera and laser radar of the driving platform are jointly calibrated; the image of the surrounding ice and snow environment is acquired by using the jointly calibrated camera; visible ice crack detection is performed based on the image to obtain visible ice crack detection results; point cloud data of the surrounding ice and snow environment is acquired by using the jointly calibrated laser radar; invisible ice crack detection is performed based on the point cloud data to obtain invisible ice crack detection results; the visible ice crack detection results and invisible ice crack detection results are converted to the driving platform coordinate system.

[0058] In this way, the advantages of cameras and lidar can be combined to accurately and efficiently detect visible and invisible ice cracks in the ice and snow environment where the driving platform is located, thereby optimizing the navigation of the driving platform in the ice and snow environment.

[0059] In conjunction with the accompanying drawings, a camera and laser radar-based ice crack detection method, device, equipment and storage medium provided by an embodiment of the present invention are described in detail through specific embodiments.

[0060] Figure 1 A flowchart of an ice crack detection method based on a camera and a laser radar is provided in an embodiment of the present invention, such as Figure 1 As shown, the ice crack detection method 100 may include the following steps:

[0061] S110, jointly calibrate the camera and lidar of the driving platform.

[0062] In some embodiments, the Autoware tool can be used to jointly calibrate the camera and lidar of a driving platform (such as a mobile robot or an intelligent vehicle) to obtain the camera's intrinsic parameter matrix (K) and the extrinsic parameter matrix (T) between the camera and the lidar.

[0063] Among them, the intrinsic parameter matrix (K) involves multiple parameters of the camera, including the focal length, principal point coordinates and distortion parameters of the camera. It is used to establish a mapping relationship between the internal information of the camera so that the camera data can be accurately interpreted and used; the extrinsic parameter matrix (T) describes the rotation and translation transformation from the camera coordinate system to the lidar coordinate system, which is used to determine the posture relationship between the camera and the lidar.

[0064] S120, using the jointly calibrated camera to acquire an image of the surrounding ice and snow environment.

[0065] In some embodiments, the jointly calibrated camera may be used to photograph the surrounding ice and snow environment to obtain an image of the surrounding ice and snow environment, such as a visual RGB image.

[0066] S130, performing visible ice crack detection according to the image to obtain a visible ice crack detection result.

[0067] Among them, visible ice cracks refer to ice cracks that can be detected from a physical perspective through images, such as cracks, ice cliffs, and ice pits. In visible ice cracks, there is a significant height difference with the surrounding environment.

[0068] In some embodiments, the image may be input into a visible ice crack detection model, and the visible ice crack detection model may detect the image to obtain a visible ice crack detection result.

[0069] The visible ice crack detection model used here can be obtained through the following steps:

[0070] The open source model of pavement crack detection on the Roboflow platform was transferred to learn multiple images of ice and snow environments with visible ice cracks marked, and a visible ice crack detection model was obtained.

[0071] For example, taking the semantic segmentation model as an example, Figure 2 Part (a) shows the curves of the transfer learning process of the semantic segmentation model, with an mAP of 90.6%. Figure 3 The semantic segmentation results of visible ice cracks are shown. For the target detection model, transfer learning is performed on it, and the results obtained are mAP = 98.7%, precision P = 97.6%, and recall R = 96.9%. The various curves in the process of transfer learning of the target detection model are as follows Figure 2 As shown in part (b) of .

[0072] S140, using the jointly calibrated laser radar to obtain point cloud data of the surrounding ice and snow environment.

[0073] In some embodiments, the jointly calibrated laser radar can be used to scan the surrounding ice and snow environment to obtain point cloud data of the surrounding ice and snow environment.

[0074] S150, performing invisible ice crack detection according to the point cloud data to obtain invisible ice crack detection results.

[0075] Among them, invisible ice cracks refer to ice cracks that cannot be detected by images from a physical perspective. Invisible ice cracks are often invisible to the naked eye. For example, ice cracks under a thin layer of ice, ice cracks blocked by objects, and ice cracks covered by snow. Here we focus on detecting invisible ice cracks covered by snow.

[0076] In some embodiments, assuming that during the process of detecting invisible ice cracks, the laser radar and the surrounding environment do not move relative to each other, multiple frames of point cloud data within a preset time length can be superimposed to obtain relatively dense point cloud data, ensuring that each position in the surrounding environment is fully scanned by the laser radar.

[0077] The region growing algorithm is used to extract the largest connected area in the superimposed point cloud data and regard it as the environmental ground. It should be noted that since the ice and snow environment is an unstructured environment, the extracted ground is usually uneven.

[0078] Extract point cloud data with a vertical height lower than or equal to the vertical height threshold in the maximum connected area. For example, use the plane grid method to extract the lowest point cloud data at the vertical height in the maximum connected area, so as to exclude the point cloud data of vertical obstacles in the point cloud data and retain the point cloud data belonging to the ground.

[0079] The extracted point cloud data is divided into multiple annular sectors of equal width and angle, which can be specifically divided into Figure 4 As shown, points of different colors represent point cloud data with different reflectivity.

[0080] Since the distance between the point cloud data in the same annular sector and the laser radar is generally similar, the reflectivity characteristics of the point cloud data in the annular sector are analyzed based on this. Specifically, for any annular sector, the reflectivity average value (f1) of the point cloud data in the annular sector is calculated, and the reflectivity threshold (f0=1.2*f1) is set based on this, and the point cloud data in the annular sector with a reflectivity (f) greater than or equal to the reflectivity threshold (f0) is determined as reflectivity abnormal point cloud data.

[0081] The reflectivity anomaly point cloud data of each annular sector is aggregated, and the aggregated reflectivity anomaly point cloud data is clustered using the DBSCAN algorithm, and each cluster obtained by clustering is determined as an invisible ice crack, thereby obtaining an invisible ice crack detection result. For example, Figure 5 The real scene of the invisible ice cracks built in the laboratory and the detection results of the invisible ice cracks are shown. The points boxed on the right side of the figure are reflectivity anomaly point cloud data, and the clusters they constitute are the invisible ice cracks covered by snow.

[0082] S160: Convert the visible ice crack detection result and the invisible ice crack detection result to a driving platform coordinate system.

[0083] In some embodiments, the visible ice crack detection results in the pixel coordinate system can be converted to the camera coordinate system based on the camera's intrinsic parameter matrix (K) obtained by joint calibration, and the visible ice crack detection results in the camera coordinate system can be converted to the lidar coordinate system based on the extrinsic parameter matrix (T) between the camera and the lidar obtained by joint calibration. The visible ice crack detection results and the invisible ice crack detection results in the lidar coordinate system can be converted to the driving platform coordinate system (for example, the robot coordinate system or the vehicle coordinate system).

[0084] According to the embodiments of the present invention, at least the following technical effects are achieved:

[0085] The advantages of cameras and lidar can be combined to accurately and efficiently detect visible and invisible ice cracks in the ice and snow environment where the driving platform is located, thereby optimizing the navigation of the driving platform in the ice and snow environment.

[0086] It is worth noting that the ice crack detection method 100 may also include:

[0087] For any annular sector, the least squares method is used to fit a two-dimensional plane according to the point cloud data in the annular sector to obtain a fitted two-dimensional plane, wherein the equation for the two-dimensional plane fitting can be Ax+By+Cz+D=0. The heights of the point cloud data in the annular sector to the two-dimensional plane are counted, and the height variance (s) is calculated based on this. If the height variance (s) is greater than or equal to the height variance threshold (s0), it is considered that the height difference is too large, and the annular sector is determined as a non-drivable area.

[0088] Summarize all the non-drivable areas and obtain the non-drivable area detection results.

[0089] At this time, S160 can be for converting the visible ice crack detection results, the invisible ice crack detection results, and the non-drivable area detection results to the driving platform coordinate system. Specifically, according to the intrinsic parameter matrix (K) of the camera obtained by joint calibration, the visible ice crack detection results in the pixel coordinate system are converted to the camera coordinate system, according to the extrinsic parameter matrix (T) between the camera and the laser radar obtained by joint calibration, the visible ice crack detection results in the camera coordinate system are converted to the laser radar coordinate system, and the visible ice crack detection results, invisible ice crack detection results, and non-drivable area detection results in the laser radar coordinate system are converted to the driving platform coordinate system.

[0090] Furthermore, the visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results in the driving platform coordinate system can also be visualized.

[0091] The ice crack detection method 100 provided by the embodiment of the present invention is described in detail below in conjunction with a specific embodiment, as shown below:

[0092] The ice crack detection method 100 is applied to a driving platform. Here, the driving platform is taken as an example of a mobile robot. The ice crack detection method 100 is deployed and tested on the mobile robot in a laboratory environment. Figure 6 As shown. Exemplarily, the mobile robot includes a robot body, a computing unit and a sensor interface. The robot body and the computing unit can interact at high speed. In this embodiment, the computing unit can be a computer with an Ubuntu operating system. In similar embodiments, the computing unit can also be a computer with other operating systems, a Jetson series artificial intelligence development board, an edge computing platform, etc. The sensor interface is connected to the camera and laser radar on the robot body and is also connected to the computing unit. In this embodiment, the camera used is Astra Pro of Orbbec, and the laser radar used is Livox Mid-360 of Livox Technology.

[0093] At this time, the ice crack detection method 100 includes the following steps:

[0094] (1) First, use the Autoware tool to jointly calibrate the camera and lidar to obtain the camera's intrinsic parameter matrix (K) and the extrinsic parameter matrix (T) between the camera and lidar.

[0095] (2) Use the jointly calibrated camera to photograph the surrounding ice and snow environment to obtain an image of the surrounding ice and snow environment.

[0096] (3) The image is input into the visible ice crack detection model, and the visible ice crack detection model detects the image to obtain the visible ice crack detection result.

[0097] (4) Use the jointly calibrated lidar to scan the surrounding ice and snow environment to obtain point cloud data of the surrounding ice and snow environment.

[0098] (5) Multiple frames of point cloud data within a preset time length are superimposed, and the maximum connected area in the superimposed point cloud data is extracted using a region growing algorithm. The point cloud data whose vertical height is less than or equal to a vertical height threshold in the maximum connected area is extracted using a plane grid method, and the extracted point cloud data is divided into multiple annular sectors of equal width and angle.

[0099] (6) For any annular sector, the least squares method is used to fit a two-dimensional plane based on the point cloud data in the annular sector to obtain a fitted two-dimensional plane. The height of the point cloud data in the annular sector to the two-dimensional plane is counted, and the height variance (s) is calculated based on this. If the height variance (s) is greater than or equal to the height variance threshold (s0), it is considered that the height difference is too large and the annular sector is determined as a non-drivable area; the height in the annular sector is positive, and the absolute value of the height (|h|) is greater than or equal to the first The point cloud data with a height threshold (h1) is determined as high point cloud data, and the point cloud data with a negative height in the annular sector and a height absolute value (|h|) less than or equal to a second height threshold (h2) is determined as low point cloud data; the reflectivity average value (f1) of the point cloud data in the annular sector is calculated, and the reflectivity threshold (f0=1.2*f1) is set based on this, and the point cloud data with a reflectivity (f) greater than or equal to the reflectivity threshold (f0) in the annular sector is determined as reflectivity abnormal point cloud data.

[0100] (7) Summarize all non-drivable areas to obtain non-drivable area detection results; summarize the high point cloud data, low point cloud data, and reflectivity anomaly point cloud data of each annular sector, and use the DBSCAN algorithm to cluster the summarized high point cloud data, low point cloud data, and reflectivity anomaly point cloud data respectively. For the high point cloud data, determine the clustered clusters as high point areas to obtain high point area detection results; for the low point cloud data, determine the clustered clusters as low point areas to obtain low point area detection results; for the reflectivity anomaly point cloud data, determine the clustered clusters as invisible ice cracks to obtain invisible ice crack detection results.

[0101] (8) According to the camera's intrinsic parameter matrix (K) obtained by joint calibration, the visible ice crack detection results in the pixel coordinate system are converted to the camera coordinate system. According to the extrinsic parameter matrix (T) between the camera and the lidar obtained by joint calibration, the visible ice crack detection results in the camera coordinate system are converted to the lidar coordinate system. The visible ice crack detection results, invisible ice crack detection results, non-drivable area detection results, high point area detection results, and low point area detection results in the lidar coordinate system are converted to the robot coordinate system.

[0102] (9) Visually display the visible ice crack detection results, invisible ice crack detection results, non-drivable area detection results, high point area detection results, and low point area detection results in the robot coordinate system. Figure 7 shown.

[0103] In this embodiment, the advantages of cameras and lidars can be combined to accurately and efficiently detect visible ice cracks, invisible ice cracks, non-drivable areas, high points, and low points on unstructured ice and snow roads in the ice and snow environment (polar environment) where the driving platform is located, thereby optimizing the navigation of the driving platform in the ice and snow environment.

[0104] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0105] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.

[0106] Figure 8 A structural diagram of an ice crack detection device based on a camera and a laser radar provided in an embodiment of the present invention, such as Figure 8 As shown, the ice crack detection device 800 may include:

[0107] The calibration module 810 is used to jointly calibrate the camera and lidar of the driving platform.

[0108] The acquisition module 820 is used to acquire images of the surrounding ice and snow environment using the jointly calibrated camera.

[0109] The detection module 830 is used to perform visible ice crack detection according to the image to obtain visible ice crack detection results.

[0110] The acquisition module 820 is used to acquire point cloud data of the surrounding ice and snow environment using the jointly calibrated laser radar.

[0111] The detection module 830 is used to perform invisible ice crack detection according to the point cloud data to obtain invisible ice crack detection results.

[0112] The conversion module 840 is used to convert the visible ice crack detection results and the invisible ice crack detection results into the driving platform coordinate system.

[0113] Understandably, Figure 8The various modules / units in the camera and laser radar based ice crack detection device 800 shown have the following functions: Figure 1 The functions of each step in the camera and lidar based ice crack detection method 100 shown can achieve its corresponding technical effects, and for the sake of brevity, they will not be repeated here.

[0114] Fig. 9 900 is a block diagram of an exemplary electronic device capable of implementing an embodiment of the present invention. Electronic device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 900 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required in the present invention.

[0115] like Fig. 9 As shown, the electronic device 900 may include a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 may also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0116] Multiple components in the electronic device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer program product, including a computer program, which is tangibly contained in a computer-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).

[0118] The various embodiments described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of the present invention, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] It should be noted that the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in an embodiment of the present invention. For the sake of concise description, they will not be repeated here.

[0122] In addition, the present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method 100 is implemented.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and the present invention is not limited here.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An ice crack detection method based on camera and laser radar, characterized in that: The method comprises: Jointly calibrate the camera and lidar of the driving platform; Use the jointly calibrated camera to obtain images of the surrounding ice and snow environment; Perform visible ice crack detection according to the image to obtain a visible ice crack detection result; Use the jointly calibrated LiDAR to obtain point cloud data of the surrounding ice and snow environment; Perform invisible ice crack detection according to the point cloud data to obtain invisible ice crack detection results; The visible ice crack detection result and the invisible ice crack detection result are converted to a driving platform coordinate system.

2. The method according to claim 1, characterized in that The joint calibration of the camera and the laser radar of the driving platform includes: The Autoware tool is used to jointly calibrate the camera and lidar of the driving platform.

3. The method according to claim 1, characterized in that The performing visible ice crack detection according to the image to obtain a visible ice crack detection result includes: Inputting the image into a visible ice crack detection model, and using the visible ice crack detection model to detect the image to obtain a visible ice crack detection result; The visible ice crack detection model is obtained by the following steps: The open source model of pavement crack detection on the Roboflow platform was transferred to learn multiple images of ice and snow environments with visible ice cracks marked, and a visible ice crack detection model was obtained.

4. The method according to claim 1, characterized in that: The step of performing invisible ice crack detection according to the point cloud data to obtain invisible ice crack detection results includes: Overlay multiple frames of point cloud data within a preset time length; Extract the largest connected area in the superimposed point cloud data; Extract point cloud data whose vertical height in the maximum connected area is lower than or equal to the vertical height threshold; Divide the extracted point cloud data into a plurality of annular sectors of equal width and equal angle; For any annular sector, the reflectivity average value of the point cloud data in the annular sector is calculated, and a reflectivity threshold is set based on the average value, and the point cloud data in the annular sector whose reflectivity is greater than or equal to the reflectivity threshold is determined as reflectivity abnormal point cloud data; The reflectivity anomaly point cloud data of each annular sector are summarized, and the summarized reflectivity anomaly point cloud data are clustered, and each cluster obtained by clustering is determined as an invisible ice crack to obtain an invisible ice crack detection result.

5. The method according to claim 4, characterized in that The method further comprises: For any annular sector, a two-dimensional plane is fitted according to the point cloud data in the annular sector to obtain a fitted two-dimensional plane, and the height from the point cloud data in the annular sector to the two-dimensional plane is counted, and the height variance is calculated based on the height variance. If the height variance is greater than or equal to the height variance threshold, the annular sector is determined as a non-drivable area; Summarize all non-drivable areas and obtain the non-drivable area detection results; The converting the visible ice crack detection result and the invisible ice crack detection result into a driving platform coordinate system includes: The visible ice crack detection result, the invisible ice crack detection result and the non-drivable area detection result are converted to a driving platform coordinate system.

6. The method according to claim 5, characterized in that The converting the visible ice crack detection result, the invisible ice crack detection result, and the non-drivable area detection result into a driving platform coordinate system includes: According to the intrinsic parameter matrix of the camera obtained by joint calibration, the visible ice crack detection result in the pixel coordinate system is converted to the camera coordinate system; According to the external parameter matrix between the camera and the laser radar obtained by joint calibration, the visible ice crack detection result in the camera coordinate system is converted to the laser radar coordinate system; The visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results in the lidar coordinate system are converted to the driving platform coordinate system.

7. The method according to claim 6, characterized in that The method further comprises: The visible ice crack detection results, invisible ice crack detection results and non-drivable area detection results in the driving platform coordinate system are visualized.

8. An ice crack detection device based on camera and laser radar, characterized in that: The device comprises: The calibration module is used to jointly calibrate the camera and lidar of the driving platform; An acquisition module is used to acquire images of the surrounding ice and snow environment using the jointly calibrated camera; A detection module, used to perform visible ice crack detection according to the image to obtain a visible ice crack detection result; The acquisition module is used to acquire point cloud data of the surrounding ice and snow environment using the jointly calibrated laser radar; The detection module is used to perform invisible ice crack detection according to the point cloud data to obtain invisible ice crack detection results; A conversion module is used to convert the visible ice crack detection result and the invisible ice crack detection result into a driving platform coordinate system.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images