Dirt detection method and device for laser radar and automatic driving vehicle
By detecting the hollow area and obstacles on the lidar point cloud data, we can determine whether the lidar lens is dirty, which solves the problem of perception ability degradation caused by the lidar dirt and improves the safety of autonomous vehicles.
Patent Information
- Application Number
- CN202411095312.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
Lidar may appear dirty in rainy days or leaves, resulting in deterioration of perception and affecting the safety of autonomous vehicles.
By obtaining the range view based on the point cloud data collected by the lidar, we judge whether there is an empty area that is not covered by obstacles, and determine whether there is dirt in the lidar lens. The method includes obtaining the first point cloud data, identifying the hollow area, performing obstacle detection, and determining whether the lens is dirty based on the obstacle information.
It improves the accuracy of lidar dirty detection, avoids mis-checking caused by transparent obstacles or open environments, and ensures the safe operation of autonomous vehicles.
Smart Images

Figure CN119068188A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of autonomous driving and smart transportation technology, and specifically to a dirt detection method, device, electronic device, computer-readable storage medium, computer program product, autonomous driving vehicle, and edge computing device for laser radar. Background Art
[0002] In an autonomous driving system that uses LiDAR for perception, there may be degradation of perception due to LiDAR contamination. For example, if there is water or mud on the surface of the LiDAR lens on a rainy day, or leaves fall on the surface of the LiDAR lens in autumn, it will affect the LiDAR's perception of the surrounding environment. In this case, the perception algorithm cannot perceive the content of obstacles in the area blocked by dirt, which will cause huge safety risks for autonomous driving.
[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention
[0004] The present disclosure provides a dirt detection method, apparatus, electronic device, computer-readable storage medium, computer program product, autonomous driving vehicle, and edge computing device for laser radar.
[0005] According to one aspect of the present disclosure, a dirt detection method for a laser radar is provided, the method comprising: acquiring a first range view based on first point cloud data collected by the laser radar; in response to determining that the first range view contains at least one first hole area, judging whether the at least one first hole area includes at least one second hole area not covered by an obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixel points having pixel values less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; and in response to determining that the at least one first hole area includes at least one second hole area not covered by the obstacle area, determining whether the lens of the laser radar is dirty based on the at least one second hole area.
[0006] According to another aspect of the present disclosure, a dirt detection device for a laser radar is provided, the device comprising: a first acquisition unit, configured to acquire a first range view based on first point cloud data collected by the laser radar; a first judgment unit, configured to judge, in response to determining that the first range view contains at least one first hole area, whether the at least one first hole area includes at least one second hole area not covered by the obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixel points whose pixel values are less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; and a first determination unit, configured to determine, in response to determining that the at least one first hole area includes at least one second hole area, whether the lens of the laser radar is dirty based on the at least one second hole area.
[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the dirt detection method for laser radar of the present disclosure.
[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the dirt detection method for laser radar of the present disclosure.
[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the dirt detection method for laser radar of the present disclosure when executed by a processor.
[0010] According to another aspect of the present disclosure, there is provided an autonomous driving vehicle, comprising: at least one laser radar; and the above-mentioned electronic device.
[0011] According to another aspect of the present disclosure, an edge computing device is provided, including: at least one laser radar; and the above-mentioned electronic device.
[0012] According to one or more embodiments of the present disclosure, the accuracy of laser radar dirt detection can be improved.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.
[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;
[0016] Figure 2 A flow chart of a dirt detection method for a laser radar according to an embodiment of the present disclosure is shown;
[0017] Figure 3 A flowchart for determining whether a laser radar lens is dirty according to an embodiment of the present disclosure is shown;
[0018] Figure 4 A flowchart of determining whether a suspected occlusion area is included in a first range view according to an embodiment of the present disclosure is shown;
[0019] Figure 5 shows a range view and a road image at a corresponding position and viewing angle according to an exemplary embodiment of the present disclosure;
[0020] Figure 6 A structural block diagram of a dirt detection device for a laser radar according to an embodiment of the present disclosure is shown;
[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0023] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.
[0024] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.
[0025] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
[0026] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .
[0027] In an embodiment of the present disclosure, the motor vehicle 110 may include a computing device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.
[0028] The server 120 may run one or more services or software applications that can perform the laser radar dirt detection method. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
[0029] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
[0030] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
[0031] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.
[0032] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, WiFi), and / or any combination of these and other networks.
[0033] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.
[0034] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.
[0035] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.
[0036] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth of the IEEE802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.
[0037] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of a computing device on the motor vehicle side (client) according to the present disclosure.
[0038] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.
[0039] According to the embodiments of the present disclosure, Figure 2 As shown, a dirt detection method for a laser radar is provided, the method comprising: step S201, acquiring a first range view based on first point cloud data collected by the laser radar; step S202, in response to determining that the first range view contains at least one first hole area, judging whether the at least one first hole area includes at least one second hole area not covered by the obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixel points whose pixel values are less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; and step S203, in response to determining that the at least one first hole area includes at least one second hole area, determining whether the lens of the laser radar is dirty based on the at least one second hole area.
[0040] Optionally, the above-mentioned laser radar can be a laser radar deployed on a vehicle, or a laser radar deployed on an edge computing device (such as a roadside device), which is not limited here. The following will take the laser radar deployed on a vehicle as an example to describe the above method in detail.
[0041] LiDAR (Light Detection and Ranging) is a device that uses laser as a signal wave for detection. It emits a laser beam and receives the signal reflected from the target to calculate the target's distance, direction, height, speed and other information. It is an important sensor in vehicles, especially in self-driving vehicles. It uses laser beams to detect the position, speed and other characteristic quantities of the target, providing the driving system with accurate environmental perception capabilities.
[0042] The laser radar emits a laser beam to the surroundings through the laser transmitting circuit. When the laser beam hits the target, the target reflects the laser back to the laser receiving circuit. The receiving circuit converts the optical signal into an electrical signal, and the main control module calculates and processes the signal to obtain a large number of discrete spatial coordinate points without topological structure, namely point cloud data. The first point cloud data mentioned above is also a frame of point cloud data collected by the laser radar on the vehicle to be used for dirt detection at a certain moment.
[0043] Range View is a compact and information-rich two-dimensional data representation generated by discretizing the point cloud data according to its angles in the horizontal and vertical directions, mapping it to a two-dimensional plane, and representing it in the form of a multi-channel image, each channel may contain characteristics such as distance and reflectivity. The first range view is generated based on the first point cloud data in the above manner.
[0044] Under normal circumstances, the laser radar is uniformly emitted to the surrounding environment, that is, in the range image obtained by the reflected light, the pixel value of each pixel (for example, it can be determined based on the reflectivity) is not zero. Therefore, when a hole area is detected on the range image (for example, the pixel value of each pixel in a connected area on the range image is less than a preset pixel value or 0), it is considered that the laser radar may be dirty.
[0045] In some scenarios, the surrounding obstacles may have a large number of transparent parts (such as large glass windows on buses), which cannot reflect lasers. Therefore, the lidar cannot receive reflected light and produces a hole area in the range view. In this case, it is very likely to cause false detection by only detecting the hole area, which will affect the normal operation of the vehicle.
[0046] In some embodiments, a method for determining whether the first range view includes at least one first hole area can be: setting a preset pixel threshold, comparing the pixel value of each pixel in the first range view with the threshold, and taking the pixel points below the threshold as part of the hole area, thereby obtaining at least one first hole area.
[0047] In some embodiments, the method of determining that the first range view includes at least one first hole area may also be to identify and detect the first range view through a trained deep learning model (such as a convolutional neural network CNN) to output at least one first hole area in the first range view. The above-mentioned deep learning model may be trained based on sample range views with marked hole areas.
[0048] In some embodiments, in response to determining that the first range view does not include a hole area, hole area detection may continue to be performed on a range view generated by a next frame of point cloud data.
[0049] In some embodiments, in response to determining that the first range view contains at least one first hole area, obstacle detection can be performed and the obstacle area on the first range view can be obtained in the following manner: clustering is performed on the first point cloud data after preprocessing to obtain the center points of one or more obstacles; and the center point is projected into the first range view according to the coordinates of the center point, and an obstacle area is determined on the first range view based on the projected center point and the preset length, width or radius.
[0050] Subsequently, it can be further determined whether the at least one first hole area includes at least one second hole area not covered by the obstacle area. In response to determining that the at least one first hole area includes at least one second hole area not covered by the obstacle area, it can be determined whether the laser radar lens is dirty based on the at least one second hole area.
[0051] In some embodiments, determining whether the laser radar lens is dirty based on at least one second hole area can be: in response to determining that there is at least one second hole area in the above-mentioned first range view, it is considered that the laser radar lens is dirty.
[0052] Therefore, in response to detecting a hole area (at least one first hole area) on the first range view, obstacle detection is further performed on the frame point cloud data (first point cloud data) to obtain the obstacle area of the obstacle on the first range view, and to determine whether the first hole area contains a second hole area that is not covered by the obstacle area; if the second hole area is included, then the laser radar lens is judged based on the second hole area to determine whether there is dirt. Therefore, by performing obstacle detection on the point cloud data and introducing the obstacle information into the laser radar dirt detection, the situation in which a hole area is generated due to the presence of transparent obstacles in the surrounding area and the laser radar cannot receive reflected light, thereby causing false detection, is avoided, thereby improving the accuracy of the laser radar dirt detection.
[0053] In some embodiments, the above-mentioned dirt detection method for laser radar may also include: in response to determining that at least one first hole area is completely covered by an obstacle area, determining that there is no dirt on the lens of the current laser radar; and continuing to perform dirt detection on the next frame range view acquired based on the laser radar.
[0054] Therefore, if it is determined that at least one first hole area is completely covered by the obstacle area, the laser radar is not considered to be dirty, and the above-mentioned detection is directly performed on the range view corresponding to the next frame of point cloud data, thereby avoiding misjudgment and improving the efficiency of dirt detection.
[0055] In some embodiments, the acquisition of the above-mentioned obstacle area can also be achieved through the following operations: performing obstacle detection on the first point cloud data to obtain at least one obstacle identification frame in the first point cloud data; and based on the coordinate information of each of the at least one obstacle identification frame, projecting at least one obstacle identification frame to the first range view to obtain the obstacle area.
[0056] In some embodiments, obstacle detection on the first point cloud data may also be performed by extracting features and detecting obstacles on the first point cloud data through a trained deep learning network (such as a point cloud convolutional neural network, etc.) to output an accurate obstacle identification frame. The deep learning network may be trained based on sample point cloud data with obstacle identification frames marked.
[0057] After the obstacle identification frame is acquired in the above manner, the obstacle identification frame may be projected into the first range view based on the coordinate information corresponding to the identification frame, thereby obtaining the obstacle region in the first range view.
[0058] Therefore, by performing obstacle detection on the first point cloud data and projecting the obstacle identification frame in the point cloud data into the first range view, the obstacle area can be obtained more accurately, thereby improving the accuracy of dirt detection.
[0059] In some embodiments, the at least one second hole region may be obtained by deleting a portion of the at least one first hole region that overlaps with the obstacle region.
[0060] In some embodiments, the acquisition of the at least one second hole area can also be achieved by the following operations: updating the pixel value of each pixel point in the area overlapping with the obstacle area in at least one first hole area to the first pixel value to obtain an updated first range view, wherein the first pixel value is not less than a preset pixel threshold; and detecting the updated first range view to obtain at least one second hole area, wherein each of the at least one second hole area includes an area composed of multiple pixel points whose pixel values are less than the preset pixel threshold.
[0061] In some embodiments, after obtaining the obstacle area, the pixel value of each pixel point of the portion where the obstacle area overlaps with the first hole area can be set to the first pixel value (for example, the pixel value of the portion is set to 255 or any pixel value greater than the above-mentioned preset pixel threshold) to fill the hole area and obtain an updated first range view. Subsequently, the hole area detection can be further performed on the updated first range view to determine at least one second hole area.
[0062] Therefore, by updating the pixel value of the area where the first hole area overlaps with the obstacle area to the first pixel value and performing hole detection again on the updated first range view, the second hole area can be obtained more accurately and efficiently, further improving the accuracy of dirt detection.
[0063] In some embodiments, Figure 3 As shown, the above-mentioned determination of whether the laser radar lens is dirty based on at least one second hole area can also be achieved through the following operations: step S301, judging whether at least one second hole area includes at least one third hole area whose area is greater than a preset area threshold; step S302, in response to determining that at least one second hole area includes at least one third hole area, based on at least one third hole area, determining whether the first range view contains suspected occlusion areas where dirty occlusion may exist; and step S303, in response to the first range view containing suspected occlusion areas, determining whether the laser radar lens is dirty based on the suspected occlusion areas in the first range view.
[0064] In some embodiments, in response to determining that at least one second hole area includes at least one third hole area, determining whether the first range view contains a suspected occlusion area where dirty occlusion may exist based on the at least one third hole area can be: directly determining the at least one third hole area as a suspected occlusion area.
[0065] In some embodiments, in response to the first range view containing a suspected occlusion area, determining whether the laser radar lens is dirty based on the suspected occlusion area in the first range view can be: in response to determining that there is a suspected occlusion area in the first range view, it is considered that the laser radar lens is dirty.
[0066] In some embodiments, in response to the first range view including a suspected occlusion area, determining whether the laser radar lens is dirty based on the suspected occlusion area in the first range view may also be: in response to determining that there is an area larger than a preset area among the suspected occlusion areas in the first range view, it is considered that the laser radar lens is dirty.
[0067] Therefore, by screening the third hole area with a larger area in the second hole area and determining the suspected occlusion area therefrom, and then determining whether the lidar lens is dirty, the accuracy of dirt detection can be further improved.
[0068] In some scenarios, when the vehicle is in a relatively open environment (such as on an overpass), the laser emitted by the LiDAR may not encounter obstacles and be reflected back, causing the LiDAR to be unable to receive the reflected light and produce holes in the range view. These holes are also likely to cause false detection of dirt, thereby affecting the normal operation of the vehicle.
[0069] In some embodiments, Figure 4 As shown, in response to determining that at least one second hole area includes at least one third hole area, determining whether the first range view contains a suspected occlusion area where dirt and occlusion may exist based on at least one third hole area can also be achieved by the following operations: step S401, in response to determining that at least one second hole area includes at least one third hole area, based on the position when the laser radar collects the first point cloud data, obtaining first map information, the first map information including road information in a high-precision map within a preset range around the position; step S402, based on the first map information and the posture when the laser radar collects the first point cloud data, obtaining a road image under the corresponding perspective of the posture, wherein the road image includes a road area; and step S403, in response to determining that at least one third hole area and the road area have an overlapping area, determining the suspected occlusion area in the first range view based on the overlapping area.
[0070] In some embodiments, the position of the above-mentioned laser radar when collecting the first point cloud data can be a world coordinate obtained based on the Global Navigation Satellite System (GNSS).
[0071] In some embodiments, the first map information can obtain road information in a high-precision map within a preset range around the corresponding location from the cloud or the vehicle based on the above-mentioned location information, such as lane boundary line information, and may further include information on other traffic elements on the lane.
[0072] In some embodiments, the first map information may be converted into a road image at a corresponding viewing angle based on the position information when the laser radar collects the first point cloud data, and the road area may be marked in the road image based on the lane boundary information. The conversion may be achieved based on the transformation between coordinate systems.
[0073] Figure 5 A range view and road images at corresponding positions and viewing angles according to an exemplary embodiment of the present disclosure are shown.
[0074] like Figure 5 As shown, image 510 is a frame range view acquired by the laser radar, and image 520 is a road image corresponding to the frame range view obtained based on the above method, wherein area 521 is a road area.
[0075] In some embodiments, in response to determining that at least one third hole region and the road region have an overlapping area, determining the suspected occlusion area in the first range view based on the overlapping area may include directly determining the overlapping area as the suspected occlusion area.
[0076] In some embodiments, in response to determining that there is an overlapping area between at least one third hole area and the road area, determining the suspected occlusion area in the first range view based on the overlapping area can also be: determining a connected area in the overlapping area whose area is greater than a preset area threshold as a suspected occlusion area.
[0077] Subsequently, based on the suspected occlusion area determined above, it is possible to further determine whether the laser radar lens is dirty based on the above method.
[0078] Therefore, by further introducing road information, the corresponding road area in the first range view is obtained, and by judging whether the road area contains a hole area, it is judged whether there is a suspected occlusion area, thereby screening out the situation in which there are few surrounding obstacles in the open scene and the laser cannot be reflected, thus further improving the accuracy of dirt detection.
[0079] In some embodiments, in response to the first range view containing a suspected occlusion area, determining whether the laser radar lens is dirty based on the suspected occlusion area in the first range view can also be achieved by the following operations: in response to the first range view containing a suspected occlusion area, sequentially detecting whether each frame in a second range view of a preset number of frames after the first range view acquired by the laser radar contains the suspected occlusion area; and in response to detecting that each frame in the second range view of a preset number of frames contains the suspected occlusion area, determining that the laser radar lens is dirty.
[0080] In some embodiments, the preset number of frames may be, for example, 20 to 30 frames, which is not limited herein.
[0081] Therefore, by performing the detection of suspected occlusion areas as described above on multiple consecutive frame range views, and only when the suspected occlusion areas are included in the second range views for a preset number of consecutive frames, is it determined that the lidar lens is dirty, thereby further avoiding false detections and improving the accuracy of dirt detection.
[0082] In some embodiments, when it is determined through the method described above that the lens of the vehicle's lidar is dirty, an early warning may be issued, and the vehicle may be controlled to limit speed, brake, or perform other operations to avoid traffic accidents.
[0083] In some embodiments, Figure 6 As shown, a dirt detection device 600 for a laser radar is provided, comprising: a first acquisition unit 610, configured to acquire a first range view based on first point cloud data collected by the laser radar; a first judgment unit 620, configured to determine, in response to determining that the first range view contains at least one first hole area, whether the at least one first hole area includes at least one second hole area not covered by the obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixel points whose pixel values are less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; and a first determination unit 630, configured to determine, in response to determining that the at least one first hole area includes at least one second hole area, based on the at least one second hole area, whether the lens of the laser radar is dirty.
[0084] Among them, the operations performed by units 610 to 630 in the above-mentioned dirt detection device 600 for laser radar are similar to the operations performed by steps S201 to S203 in the above-mentioned dirt detection method for laser radar, and are not repeated here.
[0085] In some embodiments, the first determination unit may include: a first judgment subunit, configured to determine whether at least one second hole area includes at least one third hole area whose area is greater than a preset area threshold; a first determination subunit, configured to determine whether the first range view contains a suspected occlusion area where dirty occlusion may exist based on the at least one third hole area in response to determining that at least one second hole area includes at least one third hole area; and a second determination subunit, configured to determine whether the lens of the laser radar is dirty based on the suspected occlusion area in the first range view in response to the first range view containing a suspected occlusion area.
[0086] In some embodiments, the first determination subunit can be further configured to: in response to determining that at least one second hole area includes at least one third hole area based on the position when the laser radar collects the first point cloud data, obtain first map information, the first map information including road information in a high-precision map within a preset range around the position; based on the first map information and the posture of the laser radar when collecting the first point cloud data, obtain a road image under the perspective corresponding to the posture, wherein the road image includes a road area; and in response to determining that there is an overlapping area between at least one third hole area and the road area, determine a suspected occlusion area in the first range view based on the overlapping area.
[0087] In some embodiments, the second determination subunit can be further configured to: in response to the first range view containing a suspected occlusion area, sequentially detect whether each frame in the second range view based on a preset number of frames after the first range view acquired by the laser radar contains the suspected occlusion area; and in response to detecting that each frame in the second range view of a preset number of frames contains the suspected occlusion area, determine that the laser radar lens is dirty.
[0088] In some embodiments, the first judgment unit may include: a first detection subunit, configured to perform obstacle detection on the first point cloud data to obtain at least one obstacle identification frame in the first point cloud data; and a first acquisition subunit, configured to project at least one obstacle identification frame to a first range view based on the coordinate information of each of the at least one obstacle identification frame to obtain an obstacle area.
[0089] In some embodiments, the first determination unit may further include: a first updating subunit, configured to update the pixel value of each pixel point in the area overlapping the obstacle area in at least one first hole area to a first pixel value to obtain an updated first range view, wherein the first pixel value is not less than a preset pixel threshold; and a second detection subunit, configured to detect the updated first range view to obtain at least one second hole area, wherein each of the at least one second hole area includes an area composed of multiple pixel points whose pixel values are less than the preset pixel threshold.
[0090] In some embodiments, the above-mentioned dirt detection device for laser radar may also include: a second judgment unit, configured to determine that there is no dirt on the lens of the current laser radar in response to determining that at least one first hole area is completely covered by an obstacle area; and a first detection unit, configured to continue to perform dirt detection on the next frame range view acquired based on the laser radar.
[0091] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.
[0092] According to another aspect of the present disclosure, an edge computing device is also provided. Optionally, the edge computing device may include not only electronic devices but also communication components, and the electronic devices and the communication components may be integrated or separately provided. The electronic devices may obtain data from roadside sensing devices (such as roadside cameras), such as pictures and videos, to perform image and video processing and data calculation, and then transmit the processing and calculation results to the cloud control platform via the communication components.
[0093] Optionally, the edge computing device may also be a road side computing unit (RSCU). Optionally, the electronic device itself may also have the perception data acquisition function and communication function, such as an AI camera. The electronic device can directly perform image and video processing and data calculation based on the acquired perception data, and then transmit the processing and calculation results to the cloud control platform.
[0094] Optionally, the cloud control platform performs processing in the cloud to perform image and video processing and data calculation. The cloud control platform can also be called a vehicle-road collaborative management platform, V2X platform, cloud computing platform, central system, cloud server, etc.
[0095] refer to Figure 7, a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0096] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0097] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700. The input unit 706 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.
[0098] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the above-mentioned dirt detection method for laser radar. For example, in some embodiments, the above-mentioned dirt detection method for laser radar may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the above-mentioned dirt detection method for laser radar may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the above-mentioned dirt detection method for lidar in any other appropriate manner (for example, by means of firmware).
[0099] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0100] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0101] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0103] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0104] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0105] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0106] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.
Claims
1. A contamination detection method for a laser radar, the method comprising: Acquire a first range view based on the first point cloud data collected by the laser radar; In response to determining that the first range view includes at least one first hole area, determining whether the at least one first hole area includes at least one second hole area not covered by an obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixels having a pixel value less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; as well as In response to determining that the at least one first hole area includes the at least one second hole area, it is determined whether the lens of the laser radar is dirty based on the at least one second hole area.
2. The method according to claim 1, wherein: The determining whether the lens of the laser radar is dirty based on the at least one second hole area includes: Determine whether the at least one second hole region includes at least one third hole region whose area is greater than a preset area threshold; In response to determining that the at least one second hole region includes the at least one third hole region, determining whether the first range view includes a suspected occlusion region where dirt occlusion may exist based on the at least one third hole region; and In response to the first range view including a suspected occlusion area, it is determined whether the lens of the laser radar is dirty based on the suspected occlusion area in the first range view.
3. The method according to claim 2, wherein: In response to determining that the at least one second hole area includes the at least one third hole area, determining whether the first range view includes a suspected occlusion area where dirt occlusion may exist based on the at least one third hole area comprises: In response to determining that the at least one second hole area includes the at least one third hole area, acquiring first map information based on the position when the laser radar collects the first point cloud data, the first map information including road information in a high-precision map within a preset range around the position; Based on the first map information and the position and posture of the laser radar when collecting the first point cloud data, obtaining a road image at a viewing angle corresponding to the position and posture, wherein the road image includes a road area; and In response to determining that there is an overlapping area between the at least one third hole area and the road area, a suspected occlusion area in the first range view is determined based on the overlapping area.
4. The method according to claim 2 or 3, wherein: In response to the first range view including a suspected occlusion area, determining whether the lens of the laser radar is dirty based on the suspected occlusion area in the first range view includes: In response to the first range view including a suspected occlusion area, sequentially detecting whether each frame in a second range view of a preset number of frames after the first range view acquired by the laser radar includes the suspected occlusion area; and In response to detecting that each frame in the second range view of the preset number of frames includes a suspected occlusion area, it is determined that the lens of the laser radar is dirty.
5. The method according to any one of claims 1 to 4, wherein: The acquisition of the obstacle area includes: Performing obstacle detection on the first point cloud data to obtain at least one obstacle identification frame in the first point cloud data; and Based on the coordinate information of each of the at least one obstacle identification frame, the at least one obstacle identification frame is projected onto the first range view to obtain the obstacle area.
6. The method according to any one of claims 1 to 5, wherein: The acquisition of at least one second hole region comprises: Updating the pixel value of each pixel point in the area overlapping the obstacle area of the at least one first hole area to a first pixel value to obtain an updated first range view, wherein the first pixel value is not less than the preset pixel threshold; and The updated first range view is detected to obtain the at least one second hole area, wherein each of the at least one second hole area includes an area composed of a plurality of pixel points whose pixel values are less than the preset pixel threshold.
7. The method according to any one of claims 1 to 6, further comprising: In response to determining that the at least one first hole area is completely covered by the obstacle area, determining that the lens of the current laser radar is not dirty; as well as Continue to perform dirt detection on the next frame range view acquired based on the laser radar.
8. A dirt detection device for a laser radar, the device comprising: A first acquisition unit is configured to acquire a first range view based on first point cloud data collected by the laser radar; a first judgment unit, configured to judge, in response to determining that the first range view includes at least one first hole area, whether the at least one first hole area includes at least one second hole area not covered by the obstacle area, wherein each of the at least one first hole area includes an area composed of a plurality of pixels whose pixel values are less than a preset pixel threshold, and the obstacle area is determined by performing obstacle detection on the first point cloud data; as well as The first determination unit is configured to determine whether the lens of the laser radar is dirty in response to determining that the at least one first hole area includes the at least one second hole area and based on the at least one second hole area.
9. The device according to claim 8, wherein: The first determining unit includes: A first judging subunit is configured to judge whether the at least one second hole region includes at least one third hole region whose area is greater than a preset area threshold; A first determining subunit is configured to, in response to determining that the at least one second hole area includes the at least one third hole area, determine whether the first range view includes a suspected occlusion area where dirt occlusion may exist based on the at least one third hole area; and The second determining subunit is configured to determine whether the lens of the laser radar is dirty based on the suspected occlusion area in the first range view in response to the suspected occlusion area being included in the first range view.
10. The device according to claim 9, wherein: The first determining subunit is further configured to: In response to determining that the at least one second hole area includes the at least one third hole area based on the position when the laser radar collects the first point cloud data, obtaining first map information, the first map information including road information in a high-precision map within a preset range around the position; Based on the first map information and the posture of the laser radar when collecting the first point cloud data, obtaining a road image at a viewing angle corresponding to the posture, wherein the road image includes a road area; as well as In response to determining that there is an overlapping area between the at least one third hole area and the road area, a suspected occlusion area in the first range view is determined based on the overlapping area.
11. The device according to claim 9 or 10, wherein: The second determining subunit is further configured to: In response to the first range view including a suspected occlusion area, sequentially detecting whether each frame in a second range view of a preset number of frames after the first range view acquired by the laser radar includes the suspected occlusion area; as well as In response to detecting that each frame in the second range view of the preset number of frames includes a suspected occlusion area, it is determined that the lens of the laser radar is dirty.
12. The device according to any one of claims 8 to 11, wherein: The first judging unit comprises: a first detection subunit, configured to perform obstacle detection on the first point cloud data to obtain at least one obstacle identification frame in the first point cloud data; and The first acquisition subunit is configured to project the at least one obstacle identification frame onto the first range view based on the coordinate information of each of the at least one obstacle identification frame to obtain the obstacle area.
13. The device according to any one of claims 8 to 12, wherein: The first determining unit further includes: a first updating subunit, configured to update a pixel value of each pixel point in an area overlapping the obstacle area of the at least one first hole area to a first pixel value, so as to obtain an updated first range view, wherein the first pixel value is not less than the preset pixel threshold; and The second detection subunit is configured to detect the updated first range view to obtain the at least one second hole area, wherein each of the at least one second hole area includes an area composed of a plurality of pixel points whose pixel values are less than the preset pixel threshold.
14. The device according to any one of claims 8 to 13, further comprising: A second judgment unit is configured to determine that the lens of the current laser radar is not dirty in response to determining that the at least one first hole area is completely covered by the obstacle area; as well as The first detection unit is configured to continue to perform dirt detection on a next frame range view acquired based on the laser radar.
15. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.
17. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
18. An autonomous driving vehicle comprising: at least one lidar; as well as The electronic device according to claim 15.
19. An edge computing device, comprising: at least one lidar; as well as The electronic device according to claim 15.
