Shielding detection method and device, optical detection device and terminal equipment

Abstract

The embodiment of the invention provides a shielding detection method and device, an optical detection device and terminal equipment. The shielding detection method is used for performing first shielding detection on a window of an optical detection device. The occlusion detection method comprises the following steps: acquiring point cloud data; converting the point cloud data into image data; performing feature extraction on the image data to obtain feature information; and determining a first shielding state of the window based on the feature information. According to the scheme, the safety of a system applied to the optical detection device can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of optical detection technology, and in particular to an occlusion detection method and apparatus, an optical detection device, and a terminal device. Background Technology

[0002] Optical detection devices are instruments that use light as a medium to detect objects and are finding applications in an increasing number of fields. Optical detection devices may face complex environments during application, and a housing can be used to protect the main components. A window can be provided on the housing, or the entire housing can be designed as a window, allowing the laser emitted by the optical detection device to pass through the housing, and the reflected light to be received by the internal components of the optical detection device. During use, the window may be obstructed by the environment, leading to a decrease in the detection performance of the optical detection device and thus affecting the safety of the system in which it is used. Summary of the Invention

[0003] This disclosure provides an occlusion detection method and apparatus, an optical detection device, and a terminal device to improve the security of the system to which the optical detection device is applied.

[0004] In a first aspect, an occlusion detection method is provided for performing a first occlusion detection on the window of an optical detection device. The occlusion detection method includes: acquiring point cloud data; converting the point cloud data into image data; extracting features from the image data to obtain feature information; and determining the first occlusion state of the window based on the feature information.

[0005] The above occlusion detection method utilizes point cloud data acquired during the use of the optical detection device for occlusion detection. This allows for timely understanding of the window's occlusion status during operation, enabling prompt response or handling of occlusion issues and improving the security of the system using the optical detection device. Furthermore, this occlusion detection method converts point cloud data into image data, allowing for more accurate judgment of window occlusion through image feature extraction, thus improving the accuracy of occlusion detection and further enhancing the security of the system using the optical detection device.

[0006] In one implementation, point cloud data includes location information and attribute information of multiple points, and image data includes two-dimensional location and pixel value of multiple pixels; converting point cloud data into image data includes: determining the two-dimensional location of multiple pixels based on the location information of multiple points in the point cloud data; and determining the pixel value of multiple pixels based on the attribute information of multiple points in the point cloud data, wherein the attribute information includes one or both of intensity and distance.

[0007] Optionally, the position information of the multiple points includes direction information, which includes first azimuth information and second azimuth information; or, the position information of the multiple points includes three-dimensional position coordinates.

[0008] Optionally, the pixel values ​​of multiple pixels are determined based on the attribute information of multiple points in the point cloud data, including: determining a conversion coefficient based on a first intensity; and determining the pixel value of a first pixel based on a preset pixel value and the conversion coefficient, wherein the multiple pixels include the first pixel, and the first intensity is the intensity in the point cloud data corresponding to the first pixel. By establishing a relationship between pixel values ​​and intensity, the strength of the echo energy can be represented by the pixel value. This allows for the determination of whether a viewport is occluded using a two-dimensional image.

[0009] Optionally, determining the conversion coefficient based on the first intensity includes: determining the conversion coefficient based on the first intensity and the first distance, where the first distance is the distance in the point cloud data corresponding to the first pixel.

[0010] Optionally, determining the conversion coefficient includes: determining a first echo energy coefficient based on a first intensity or based on the first intensity and a first distance; adjusting the dynamic range of the first echo energy coefficient to obtain a second echo energy coefficient; and performing equalization processing on the second echo energy coefficient to obtain the conversion coefficient. Thus, by adjusting the dynamic range of the first echo energy coefficient, a more reasonable difference can be made between different echo energy coefficients, facilitating analysis and processing, and making the final pixel values ​​more able to distinguish the strength of the echo energy.

[0011] In one implementation, determining the two-dimensional positions of multiple pixels based on the positional information of multiple points in point cloud data includes: correcting the positional information of the multiple points to obtain corrected positional information; and determining the two-dimensional positions of the multiple pixels based on the corrected positional information. Thus, by correcting the positional information of multiple points, the quality of the converted image can be improved, further enhancing the accuracy of occlusion detection.

[0012] Optionally, the position information of multiple points includes the direction information of multiple points, and the correction of the position information includes the correction of the first azimuth information and the second azimuth information.

[0013] Optionally, the field of view of the optical detection device includes multiple luminescent areas, and the position information of multiple points is corrected, including: correcting the position information of multiple points based on the positions of multiple luminescent areas.

[0014] In one implementation, the above occlusion detection method further includes: acquiring first information, which indicates a second occlusion state of the viewport; and determining the occlusion status of the viewport based on the first and second occlusion states. Combining the first and second occlusion states for comprehensive occlusion judgment can improve the accuracy of occlusion detection.

[0015] Optionally, the first information is used to indicate the detection result obtained by the second occlusion detection, which obtains the second occlusion state using a detection laser or detector, or the second occlusion detection obtains the second occlusion state using stray light.

[0016] In one implementation, the above feature information includes one or more of single-frame features and inter-frame features.

[0017] Optionally, the above feature information includes one or more of the following: image information entropy, image frequency domain energy, image edge information, and image correlation coefficient.

[0018] In one implementation, determining the first occlusion state of the window based on feature information includes: determining the first occlusion state of the window based on single-frame features and inter-frame features.

[0019] Optionally, single-frame features include image information entropy, and inter-frame features include image correlation coefficients.

[0020] In one implementation, determining the first occlusion state of the window based on feature information includes: when the image information entropy within the sliding window is less than or equal to a first threshold and the image correlation coefficient is greater than or equal to a second threshold, determining that the region where the sliding window is located is occluded; merging the regions of at least two sliding windows that are occluded to obtain the region where the window is occluded.

[0021] In one implementation, the feature information includes feature information of multi-frame image data obtained by converting point cloud data from multiple scanning cycles.

[0022] In one implementation, acquiring point cloud data includes: acquiring point cloud data for multiple scanning cycles; converting the point cloud data into image data, including: converting the point cloud data for multiple scanning cycles into multi-frame image data; and extracting features from the image data to obtain feature information, including: extracting features from the multi-frame images to obtain feature information of the multi-frame images.

[0023] Optionally, the feature information of the above multi-frame images includes single-frame features and inter-frame features, where the number of inter-frame features is less than or equal to the number of frames in the image. Joint determination using multiple feature information helps improve the accuracy of occlusion detection.

[0024] In a second aspect, an occlusion detection device is provided for performing a first occlusion detection on the window of an optical detection device. The occlusion detection device includes: an acquisition unit for acquiring point cloud data; a conversion unit for converting the point cloud data into image data; an extraction unit for extracting features from the image data to obtain feature information; and a determination unit for determining the first occlusion state of the window based on the feature information.

[0025] Thirdly, an occlusion detection device is provided for performing a first occlusion detection on the window of an optical detection device. The occlusion detection device includes a processor for calling instructions stored in a memory. When the instructions are executed, the occlusion detection device performs the first aspect or any one of the occlusion detection methods described above.

[0026] Fourthly, an occlusion detection system is provided, comprising: an optical detection device for acquiring sensing data, the sensing data including point cloud data or used to determine point cloud data; and an occlusion detection device communicatively connected to the optical detection device for performing the occlusion detection method described in the first aspect or any one of the first aspects.

[0027] Fifthly, a terminal device is provided, including the occlusion detection system described in the fourth aspect above.

[0028] In a sixth aspect, an optical detection device is provided, comprising a housing having a viewing window; a laser disposed within the housing for emitting laser light through the viewing window; a detector disposed within the housing for receiving laser echoes through the viewing window and converting the echoes into electrical signals; and a processor disposed within the housing for executing the first aspect or any of the occlusion detection methods described above, wherein point cloud data is obtained based on electrical signals.

[0029] In a seventh aspect, a computer-readable storage medium is provided, including instructions stored thereon, wherein, when the instructions are executed by a processor, the first aspect or any of the occlusion detection methods in the first aspect is executed. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the following description of the embodiments will be provided as examples. The drawings described below are merely embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain this disclosure and do not constitute a limitation of this disclosure.

[0031] Figure 1 The illustration shows an application scenario diagram of an optical detection device provided in some embodiments of this disclosure.

[0032] Figure 2 A schematic diagram of the structure of some optical detection devices provided in some embodiments of this disclosure is shown.

[0033] Figure 3 A schematic diagram of the main internal structure of an optical detection device provided in some embodiments of this disclosure is shown.

[0034] Figure 4 A schematic diagram of the structure of an occlusion detection system provided in some embodiments of this disclosure is shown.

[0035] Figure 5 A schematic diagram of another occlusion detection system provided in some embodiments of this disclosure is shown.

[0036] Figure 6 A flowchart illustrating an occlusion detection method provided in some embodiments of this disclosure is shown.

[0037] Figure 7 A schematic diagram of the direction information of a point provided in some embodiments of this disclosure is shown.

[0038] Figure 8 A schematic diagram of a process for converting point cloud data into image data is shown in some embodiments of this disclosure.

[0039] Figure 9 A flowchart illustrating another occlusion detection method provided in some embodiments of this disclosure is shown.

[0040] Figure 10 A schematic diagram of the arrangement of light-emitting areas provided in some embodiments of this disclosure is shown.

[0041] Figure 11 A flowchart illustrating another occlusion detection method provided in some embodiments of this disclosure is shown.

[0042] Figure 12 A schematic diagram of an occlusion detection structure provided in some embodiments of this disclosure is shown.

[0043] Figure 13 A schematic diagram of the waveform of a detector provided in some embodiments of this disclosure is shown.

[0044] Figure 14 A schematic diagram of the structure of an occlusion detection device provided in some embodiments of this disclosure is shown.

[0045] Figure 15 A schematic diagram of another occlusion detection device provided in some embodiments of this disclosure is shown. Detailed Implementation

[0046] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the specific implementation methods of this disclosure will be described below with reference to the accompanying drawings. The accompanying drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without creative effort. Adjustments and improvements made without departing from the concept of this disclosure are all within the protection scope of this disclosure.

[0047] To keep the drawings simple, each figure only schematically shows the parts related to the corresponding embodiment, and they do not represent the actual structure of the product. In addition, for the sake of simplicity and ease of understanding, some figures only schematically show parts of components with the same structure or function, and there may actually be more or fewer components with the same structure or function.

[0048] In this disclosure, unless otherwise expressly specified and limited, ordinal numbers, such as “first,” “second,” etc., are used only to distinguish and describe related objects and should not be construed as indicating or implying the relative importance or order between related objects; furthermore, they do not represent the quantity of related objects. “Multiple” includes two or more, and other quantifiers are similar. “And / or” is used to describe the relationship between related objects, which includes any combination relationship between related objects, such as “a and / or b” including: “a alone,” “b alone,” or “a and b.” “One or more of multiple objects” refers to any object or any combination of multiple objects, such as “one or more of a1, a2, a3” including: “a1 alone,” “a2 alone,” “a3 alone,” “a1 and a2,” “a1 and a3,” “a2 and a3,” or “a1, a2 and a3.”

[0049] An optical detection device is a device that uses light as a medium to detect objects. Lasers, compared to ordinary light sources, possess characteristics such as monochromaticity and good directionality, making them ideal for object detection. During object detection, the optical detection device emits a laser beam. Upon encountering an object, the laser beam is reflected from its surface, and a portion of the reflected light is received by the optical detection device. This received reflected light is converted into an electrical signal within the device, and information about the object is obtained through processing of these signals. This information may include the object's distance, position, velocity, or even its three-dimensional structure.

[0050] Optical detection devices have found applications in many fields, such as intelligent driving (or autonomous driving), drones, geographic mapping, and environmental monitoring. These devices can be installed on terminal equipment to provide sensing information, enabling the equipment to understand its environment. Examples of such optical detection devices include LiDAR (light detection and ranging). Terminal equipment can include vehicles, drones, industrial robots, mapping terminals, or monitoring terminals.

[0051] Please refer to Figure 1 This illustration shows a schematic diagram of an application scenario for an optical detection device provided in some embodiments of this disclosure. For example... Figure 1As shown, the optical detection device is, for example, a lidar 110, and the terminal device is, for example, a vehicle 120. The lidar 110 is mounted on the vehicle 120 and is used to detect objects around the vehicle 120. Figure 1 The lidar 110 is mounted on the front side of the roof, which is only an example. The dashed box in the figure also shows several other possible mounting locations. For example, the mounting locations of lidar 110 on vehicle 120 include, but are not limited to: the top of the vehicle (e.g., the front side, rear side, or other roof locations), near the headlights, hidden inside the headlights, on both sides of the body, on the front bumper, inside the grille, above the windshield, above the rear window, on the side fenders, at the front wheel arches, on the hood, on the trunk lid, under the trunk, or in a location inside the cabin that can be detected from the outside through the windshield. One or more lidars 110 may be mounted on vehicle 120. This disclosure does not limit the number and location of lidars installed on the vehicle. The number and location of lidars can be selected and designed based on various factors such as perception performance, safety, cost, appearance, and engineering. For example, a lidar can be installed on the front of the roof; another example is to install a lidar near the headlights on both sides of the vehicle; yet another example is to install a lidar on the front of the roof and a lidar on each side of the fender; yet another example is to install a lidar at the front and rear of the roof and a lidar on each side of the front wheel arch.

[0052] This disclosure does not limit the type of lidar. For example, the lidar can be mechanical, semi-solid-state, or solid-state. Semi-solid-state lidar can be, for example, a microelectromechanical system (MEMS) lidar, a rotating mirror lidar, or a prism lidar. Solid-state lidar can be, for example, an optical phase array (OPA) lidar or a flash lidar. When a vehicle is equipped with multiple lidars, the types and shapes of the lidars can be the same or different.

[0053] Optical detection devices may face complex environments in applications, and a housing can protect the main components. A viewing window can be provided on the housing, or the entire housing of the optical detection device can be designed as a viewing window, allowing the laser emitted by the optical detection device to pass through the housing and be reflected by the internal components of the optical detection device. For example, please refer to... Figure 2 This illustrates a schematic diagram of the structure of some optical detection devices provided in some embodiments of this disclosure. For example... Figure 2As shown in (a), the optical detection device 200 includes a housing 210, on which a window 220 (or filter, etc.) is provided. The window 220, as part of the housing 210, can protect the internal components; for example, by physically isolating the internal and external environments of the optical detection device 200, so that the internal components are not affected by the external environment. Figure 2 (a) This disclosure is by way of example only and does not limit the shape of the housing 210 and the window 220; for example, other shapes may also be used. Figure 2 (b)- Figure 2 Any of the shapes or structures in (d) may be used, in addition to other designs.

[0054] The main components of the optical detection device are housed inside the casing 210. Please refer to [reference needed]. Figure 3 This illustrates a schematic diagram of the main internal structure of an optical detection device provided in some embodiments of this disclosure. For example... Figure 3 As shown, the optical detection device 300 includes a laser emitting system 310, a laser receiving system 320, and a control and processing system 330. Optionally, the optical detection device 300 may also include a scanning system 340; for example, a mechanical lidar or a semi-solid-state lidar may also include a scanning system 340. The scanning system 340 may include a rotating optical engine, a rotating mirror, a tilting mirror, a galvanometer, or other components that can direct the laser beam to different locations in the environment. The laser emitting system 310, the laser receiving system 320, the control and processing system 330, and the scanning system 340 may be housed inside the housing 210 for operation under the protection of the housing 210.

[0055] The laser emitting system 310 can be used to emit laser light. When the laser light encounters object 10, it is reflected from the surface of object 10 to the optical detection device 300. The laser receiving system 320 receives the reflected echo and converts it into an electrical signal, which is then provided to the control and processing system 330. The control and processing system 330 receives the electrical signal and processes it to obtain information about the object, such as its distance, position, or velocity, as well as its three-dimensional structure. The laser emitting system 310 mainly includes a driving circuit (also called an excitation source), a laser, and emitting optical elements. Driven by the driving circuit, the laser emits light, which passes through the emitting optical elements and is emitted through the window of the optical detection device to the outside of the device.

[0056] The laser receiving system 320 mainly includes receiving optical elements and a detector. The receiving optical elements collect the echo reflected from the object and focus it onto the photosensitive surface of the detector; the detector uses the photoelectric effect to convert the optical signal into an electrical signal. The optical detection device 300 may also include a preprocessing circuit. This preprocessing circuit may include digitization circuitry, such as an analog-to-digital converter (ADC), which can convert analog signals into digital signals and provide them to the control and processing system 330. Alternatively, the preprocessing circuit may include a time-to-digital converter (TDC). When the control and processing system 330 controls the laser to emit laser light through the drive circuit, it can synchronously send a timing signal to the TDC. Subsequently, the echo is converted into an electrical signal by the detector, and this electrical signal can be converted (e.g., amplified into a voltage and compared with a reference voltage to determine if light is incident) and provided to the TDC. Based on the received electrical signal, the TDC can time the arrival time of the echo and provide the time information to the control and processing system 330. The preprocessing circuit may also include an analog front-end circuit for channel selection and analog signal amplification. The transmitting and receiving optical elements include, for example, one or more optical elements such as lenses, mirrors, filters, and beam splitters. The transmitting and receiving optical elements can be independently configured optical elements, or they can be fully or partially reused.

[0057] The control and processing system 330 may include an information processing circuit and a light source control circuit. The information processing circuit processes electrical signals to obtain information about the object. The information processing circuit can be implemented using an application-specific integrated circuit (ASIC), or a hardware circuit implemented with a programmable logic device (PLD), such as a field-programmable gate array (FPGA), a microcontroller unit (MCU), or a digital signal processor (DSP). This implementation method is beneficial for improving the efficiency of information processing. In other implementations, the information processing circuit may also be a central processing unit (CPU). The light source control circuit sends control commands (or signals) to the drive circuit to control the drive circuit to drive the laser to emit light. Pulsed laser emission can be achieved. For example, timing signals can be sent to control the laser emission timing; furthermore, encoding functions can be added by configuring the pulse interval, pulse intensity, and pulse width. This can enhance anti-interference capabilities. The light source control circuit and the information processing circuit can be integrated together, for example, integrated into a main control chip, or they can each be independent or partially independent chips. When the optical detection device 300 includes a scanning system 340, the control and processing system 330 may also include a scanning control circuit for controlling the scanning system. The scanning control circuit can be integrated with one or all of the light source control circuit and the information processing circuit. For example, the scanning control circuit, the light source control circuit, and the information processing circuit can be integrated into a main control chip; or they can be independent chips; this disclosure does not impose any limitations. In one implementation, the control and processing system 130 can be implemented as a system-on-chip (SOC) or an ASIC.

[0058] During use, the viewing window of an optical detection device may be obstructed due to changes in the environment, or the surface of the window may be covered with obstructions or dirt. Obstructions can be contact-based or non-contact-based. Contact-based obstructions include obstructions adhering to the surface of the window. Non-contact-based obstructions include obstructions near the window that partially or completely block the view. For example, in rain, snow, or sandstorms, the surface of the optical detection device may be covered with water stains, mud, or dust. Furthermore, during use, flying insects or other organisms may collide with the window surface, leaving residue or other obstructions. Additionally, if the optical detection device is installed in a location within the cabin that allows detection through the windshield or inside a headlight housing, the windshield or headlight housing may also be covered with water stains, mud, dust, or insect residue, which may obstruct part or all of the view.

[0059] The laser emitted by the optical detection device exits through a window. If the window is obstructed, the laser may be reflected, refracted, diffracted, scattered, or absorbed by the obstruction, leading to attenuation of laser energy or dispersion of spot energy, thus reducing detection capability. Therefore, obstructions can affect the detection performance of the optical detection device. This embodiment of the present disclosure considers the above problems and performs obstruction detection on the window of the optical detection device, facilitating timely handling of obstructions to reduce their impact on the optical detection device and improve the security of the system in which the optical detection device is used.

[0060] The following description is in conjunction with the accompanying drawings.

[0061] Figure 4 A schematic diagram of the structure of an occlusion detection system provided in some embodiments of this disclosure is shown. For example... Figure 4 As shown, the system is located in terminal device 400 and includes an optical detection device 410 and an occlusion detection device 420. The optical detection device 410 is used to acquire sensing data, which includes point cloud data, or the sensing data is used to determine point cloud data. The occlusion detection device 420 is communicatively connected to the optical detection device 410 and is used to perform occlusion detection. During occlusion detection, the occlusion detection device 420 obtains sensing data from the optical detection device 410 and performs occlusion detection on the viewing window of the optical detection device based on the acquired sensing data.

[0062] In some embodiments of this disclosure, the occlusion detection device 420 may be independent of the optical detection device 410. For example, the occlusion detection device 420 may be a processing or control device of the terminal device 400. Taking the terminal device 400 as a vehicle as an example, the occlusion detection device 420 may be an on-board processing or control device. For example, the occlusion detection device 420 may include, but is not limited to: a domain control unit (DCU), an electronic control unit (ECU), a vehicle central computer (VCC), a zone controller (zonal / zone ECU, or zone control unit, ZCU), an MCU, or a vehicle control unit (VCU), etc. Domain controllers include, for example, a vehicle domain controller (VDC), a cockpit domain controller (CDC), or an advanced driving assistance system / autonomous driving (ADAS / AD) domain controller (ADC), etc.

[0063] In some embodiments of this disclosure, the occlusion detection device 420 may be integrated within the optical detection device 410, for example, as a processing or control device within the optical detection device 410. The occlusion detection device 420 may be integrated with all or part of the circuitry of the control and processing system, or may be independent of the control and processing system.

[0064] In some embodiments of this disclosure, please refer to Figure 5 This illustrates a schematic diagram of another occlusion detection system provided in some embodiments of this disclosure. For example... Figure 5As shown, the system includes an optical detection device 510 and an occlusion detection device 21. The optical detection device 510 is located in the terminal device 500, and the occlusion detection device 21 is located in the server 20. The terminal device 500 also includes a communication device 520 for communicating with the server 20. The optical detection device 510 is used to acquire sensing data. The sensing data includes point cloud data, or the sensing data is used to determine point cloud data. The optical detection device 510 sends the sensing data to the communication device 520, and the communication device 520 sends the sensing data to the server 20. The occlusion detection device 21 located in the server 20 obtains the sensing data and performs occlusion detection on the viewing window of the optical detection device based on the obtained sensing data. The optical detection device 510 can send the sensing data directly to the communication device 520, or the optical detection device 510 can send the sensing data to the controller 530 of the terminal device 500, which processes the sensing data and sends it to the communication device 520, or the controller 530 forwards the sensing data to the communication device 520. Taking a vehicle as an example as the terminal device, the optical detection device could be a lidar (Light Detection and Ranging) system. Lidar can directly transmit the perceived data to the communication device via the vehicle's bus. Alternatively, lidar can transmit the perceived data to the intelligent driving domain controller or the onboard central computer, which then processes the data and sends it to or forwards it to the communication device.

[0065] Communication between the optical detection device 510 and the communication device 520, between the optical detection device 510 and the controller 530, and between the controller 530 and the communication device 520 can be achieved through wired or wireless communication methods. Wired communication methods include, but are not limited to: controller area network (CAN) bus, CAN with flexible data rate (CAN FD) bus, local interconnect network (LIN) bus, FlexRay bus, media-oriented systems transport (MOST) bus, low voltage differential signaling (LVDS) bus, time-triggered protocol / class C (TTP / C) bus, or Ethernet bus; wireless communication methods include, but are not limited to: wireless sensor networks (e.g., Bluetooth, StarFlash, or ZigBee), wireless local area networks (e.g., WLAN or WiFi), short-range point-to-point communication, near field communication (NFC), etc. Communication between the communication device 520 and the server 20 can be achieved through wireless communication, such as, but not limited to, cellular networks (e.g., 3G, 4G, 5G, or 6G). Furthermore, the communication methods between the optical detection device 510 and the communication device 520, between the optical detection device 510 and the controller 530, and between the controller 530 and the communication device 520 can be the same or different.

[0066] The occlusion detection method is described below with reference to the attached diagram.

[0067] Please refer to Figure 6 This illustration shows a flowchart of an occlusion detection method provided in some embodiments of this disclosure. The method is executed by an occlusion detection device, which may be located in a terminal device, a server, or an optical detection device. Figure 6 As shown, the occlusion detection method is used to perform a first occlusion detection on the window of an optical detection device, and includes at least the following steps:

[0068] S610: Acquire point cloud data;

[0069] S620: Converts point cloud data into image data;

[0070] S630: Perform feature extraction on image data to obtain feature information;

[0071] S640: Determine the first occlusion state of the window based on feature information; or, obtain the first occlusion detection result based on feature information.

[0072] Optical detection devices can acquire point cloud data during use. The above occlusion detection method utilizes this point cloud data to detect occlusion, allowing for timely understanding of window occlusion and prompt response or handling, thus improving the security of the system using the optical detection device. Furthermore, converting the point cloud data into image data allows for more accurate window occlusion assessment through image feature extraction, improving the accuracy of occlusion detection, reducing unnecessary processing or security issues caused by false positives, and further enhancing the security of the system using the optical detection device.

[0073] For example, when LiDAR is applied in the field of intelligent driving, its perception data is used for driving decisions, which are crucial for driving safety. If the viewport is obstructed, it will affect the LiDAR's detection performance, potentially leading to erroneous perception information. If the vehicle uses this incorrect information to make driving decisions, it could cause safety issues. The obstruction detection method described above can promptly detect obstructions in the viewport during LiDAR use, allowing for timely notification to the user to address the obstruction, automatic LiDAR cleaning, or prompting the intelligent driving system to disregard the perception data from the optical detection device, or even instructing the user to take over driving, thereby improving driving safety.

[0074] A point cloud is a collection of spatial points obtained by an optical detection device. A point cloud consists of multiple discrete points, each corresponding to a spatial location. A laser pulse emitted by the optical detection device propagates through space, encounters an object at a spatial location, and is reflected. The reflected echo is received by the optical detection device and processed to generate sensing data. This sensing data includes point cloud data or data used to determine the point cloud. Information about an object at that spatial location can be represented by a single point in the point cloud.

[0075] Data for a point can include location information and attribute information. Attribute information indicates the point's properties, such as one or more attributes like distance and intensity; intensity can also be called reflection intensity or echo intensity. Location information indicates the point's position in space. This location information may include, for example, three-dimensional position coordinates and / or orientation information. Three-dimensional position coordinates indicate the point's three-dimensional position (i.e., spatial position). Taking a Cartesian coordinate system as an example, a point's three-dimensional position in space is indicated by its three-dimensional position coordinates (x, y, z). For example, in an optical detection device coordinate system, the X-axis represents the front of the optical detection device, the Y-axis represents the left side of the optical detection device, and the Z-axis is perpendicular to the ground. Similarly, in a vehicle coordinate system, the X-axis represents the front of the vehicle, the Y-axis represents the left side of the vehicle, and the Z-axis is perpendicular to the ground. The above description of three-dimensional position coordinates using a Cartesian coordinate system is merely an example; the embodiments disclosed in this disclosure do not limit the type of coordinate system. It can be a Cartesian coordinate system, or other types of coordinate systems, such as cylindrical coordinate systems or spherical coordinate systems, as long as they can indicate the spatial position of a point in a point cloud. Directional information can be represented by angular information. For example, directional information includes a first directional angle and a second directional angle. The first directional angle may include, for example, a horizontal angle or azimuth angle, and the second directional angle may include, for example, a vertical angle or pitch angle. The first directional angle can be used to locate the direction of a point in the horizontal direction, and the second directional angle can be used to locate the direction of a point in the vertical direction. Combined with distance information, the spatial location of the point can be determined. For example, please refer to... Figure 7 This diagram illustrates the orientation information of a point provided in some embodiments of this disclosure. In the diagram, angle a represents a horizontal angle or azimuth angle, angle b represents a vertical angle or pitch angle, P represents a point in a point cloud, and d represents the distance of point P detected by the optical detection device. The horizontal angle or azimuth angle represents the angle of offset of a point relative to a horizontal direction (e.g., the X-axis direction) in the horizontal direction; the vertical angle or pitch angle represents the angle of offset of a point relative to the horizontal plane in the vertical direction. The orientation information and distance of a point can be converted between the point's three-dimensional position coordinates. The coordinate value of point P on the X-axis is x = d*COS(b)*COS(a), the coordinate value on the Y-axis is y = d*SIN(b), and the coordinate value on the Z-axis is z = d*COS(b)*SIN(a).

[0076] In some embodiments of this disclosure, the point cloud data may optionally include one or more of the following: channel information, time information, and optical detection device parameters. Channel information indicates the channel of a corresponding point, for example, through a channel identifier (ID). A channel can also be called a detection channel. For example, a channel can refer to the measurement path of a lidar. A laser emits laser light in a certain direction. When it encounters an object, it is reflected by the object's surface, received by a detector, and converted into an electrical signal; the measurement path from emission to reception can be understood as a channel. Lidar can obtain object information within its field of view by acquiring data from multiple channels. A lidar channel can correspond to a laser emission direction, which is characterized, for example, by a horizontal angle (or azimuth angle) and a vertical angle (or elevation angle). A lidar channel can correspond to a laser beam and be associated with the emission direction of that laser beam, or with the direction information of the corresponding point in the point cloud data obtained by that channel. Time information includes, for example, a timestamp. Optical detection device parameters indicate, for example, one or more of the following parameters: optical detection device model, angular resolution, field of view, frame rate, and point frequency.

[0077] During point cloud acquisition, the optical detection device manipulates a light beam to emit laser pulses in different directions in space and collects the reflected echoes. Based on the reflected echoes, electrical signals are converted and processed to obtain information such as echo intensity and distance. The optical detection device can locally convert the processed information into point cloud data according to the format and content of the point cloud data for storage; in this case, the sensing data includes point cloud data. Alternatively, the optical detection device can locally store the processed information as sensing data and send it to other devices (such as the occlusion detection device 420 and controller 530), which then convert the sensing data into point cloud data. In this case, the sensing data is used to determine the point cloud data.

[0078] The methods by which an optical detection device manipulates the light beam (scanning method) can include, for example, mechanical, semi-solid-state, and solid-state methods. The ranging methods of an optical detection device can include, for example, time-of-flight (ToF) methods, frequency-modulated continuous wave (FMCW) methods, triangulation methods, and phase methods. This disclosure does not limit the scanning method and ranging method of the optical detection device.

[0079] In step S610 above: When the occlusion detection device is located on the terminal device, the optical detection device sends the sensing data to the occlusion detection device; the occlusion detection device obtains the sensing data from the optical detection device and obtains point cloud data based on the sensing data. When the occlusion detection device is located on the server, the optical detection device sends the sensing data directly or indirectly to the communication device of the terminal device, and the communication device sends the sensing data to the occlusion detection device; the occlusion detection device obtains the sensing data from the terminal device and obtains point cloud data based on the sensing data. When the occlusion detection device is located on the optical detection device, the occlusion detection device obtains the sensing data from the storage space of the optical detection device and obtains point cloud data based on the sensing data.

[0080] In some embodiments of this disclosure, the optical detection device performs periodic scanning. Within one scanning cycle, the point cloud data acquired by the optical detection device scanning the field of view constitutes one frame of point cloud data. The scanning cycle refers to the time it takes for the laser pulse emitted by the optical detection device to cover the field of view once. The optical detection device can perform continuous or discontinuous scanning to obtain multiple frames of point cloud data across multiple scanning cycles. In some embodiments of this disclosure, the point cloud data can be converted into image data frame by frame.

[0081] The process of converting point cloud data into image data can include a point cloud mapping process and a point cloud pixelation process. In some embodiments of this disclosure, a three-dimensional point cloud can be projected onto a two-dimensional plane frame by frame to convert the point cloud data into image data, resulting in a two-dimensional image. A two-dimensional image can be understood as pixels arranged in a two-dimensional array. For example, the two-dimensional array includes n*m pixels, where n is the number of rows and m is the number of columns in the two-dimensional array. The point cloud mapping process can be understood as the process of converting point cloud data into pixel positions in a two-dimensional array. The point cloud pixelation process can be understood as the process of converting point cloud data into pixel values ​​in a two-dimensional array. Point cloud data includes the positional and attribute information of multiple points, while image data includes the two-dimensional positional information and pixel values ​​of multiple pixels. Please refer to... Figure 8 This is a schematic diagram illustrating a process for converting point cloud data into image data, provided in some embodiments of this disclosure. For example... Figure 8 As shown, step S620 above may include:

[0082] S621: Determine the two-dimensional position of multiple pixels based on the position information of multiple points in point cloud data;

[0083] S622: Determine the pixel values ​​of multiple pixels based on the attribute information of multiple points in point cloud data; the attribute information used in determining the pixel values ​​may include one or both of intensity and distance.

[0084] The location information of points in the point cloud includes, for example, orientation information and / or three-dimensional position coordinates, and the description of orientation information and three-dimensional position coordinates is the same as in the above embodiments.

[0085] In some implementations of the point cloud mapping process (S621), the two-dimensional positions of multiple pixels can be determined based on orientation information. For example, data from a frame of point cloud data can be converted into the positions of multiple pixels in a two-dimensional matrix according to horizontal angles (or azimuth angles) and vertical angles (or pitch angles). The matrix can be arranged horizontally according to the magnitude of the horizontal angles; the matrix can be arranged vertically according to the magnitude of the vertical angles. Alternatively, the matrix can be arranged vertically according to the magnitude of the horizontal angles; the matrix can be arranged horizontally according to the magnitude of the vertical angles. The vertical arrangement order can be descending or ascending; similarly, the horizontal arrangement order can be descending or ascending. For example, if an optical detection device has 128 vertical angles in its vertical field of view (e.g., a 128-beam lidar) and 1000 horizontal angles in its horizontal field of view (e.g., if the horizontal field of view of the optical detection device is 100°, scanning the entire horizontal field of view with a horizontal resolution of 0.1° yields 1000 horizontal angles), then a frame of point cloud data can be converted into image data comprising 1000*128 pixels. For example, an optical detection device may have 64 vertical angles in its vertical field of view (e.g., a 64-beam lidar) and 1800 horizontal angles in its horizontal field of view (e.g., an optical detection device with a horizontal field of view of 360°, scanning the entire horizontal field of view with a horizontal resolution of 0.2°, yields 1800 horizontal angles). In this case, a frame of point cloud data can be converted into image data comprising 1800*64 pixels. The above numbers of horizontal and vertical angles are merely examples; other values ​​may be set in other embodiments, and this disclosure does not impose any limitations.

[0086] In some implementations of the point cloud mapping process (S621), the two-dimensional positions of multiple pixels can be determined based on three-dimensional position coordinates; the point cloud data is then projected into a two-dimensional image. For example, the projection resolution can be characterized by the cuboid range (Δl*Δw*Δh) of the point cloud space, which corresponds to a pixel in the two-dimensional image, where Δl represents the length of the cuboid, Δw represents the width of the cuboid, and Δh represents the height of the cuboid. This disclosure does not limit the size of the cuboid range, i.e., it does not limit the number of pixels in the two-dimensional image. For example, by setting the sizes of Δl, Δw, and Δh, the unit projection range in the three-dimensional space is set, thereby dividing the point cloud space into multiple unit projection ranges. A correspondence between the unit projection range and the two-dimensional position coordinates of the pixels is established; during the projection process, points in the point cloud data can be traversed to determine the unit projection range to which the point belongs, and the corresponding pixel is determined using the correspondence between the unit projection range and the two-dimensional position coordinates of the pixels, thereby determining the position of the point in the point cloud as a pixel in the two-dimensional space. This disclosure does not limit the range occupied by a pixel in two-dimensional space. The range of a pixel can be characterized by Δx*Δy, where Δx represents the size occupied by a pixel in the X-axis direction and Δy represents the size occupied by a pixel in the Y-axis direction. Taking Δl as being aligned with the X-axis direction and Δw as being aligned with the Y-axis direction as an example, Δl can be the same or different from Δx, and Δw can be the same or different from Δy.

[0087] In some implementations of point cloud pixelation (S622), the pixel value of a pixel can be determined by one or both of the intensity and distance of the point cloud data. Intensity reflects the magnitude of the echo energy. Correlating intensity with the pixel value of an image can achieve image visualization effects. For example, when correlated with grayscale values, areas with high intensity appear bright, and areas with low intensity appear dark. Similarly, when correlated with color values, color rendering effects can be achieved, such as different colors appearing at locations with different intensities. For instance, a laser beam is emitted through a window, and the echo is received through the window. If the window is obstructed, the energy of the emitted laser or echo will be weakened. By establishing a relationship between pixel value and intensity, the strength of the echo energy can be represented by the pixel value. It is possible to determine whether there is obstruction in the window using a two-dimensional image.

[0088] The process of determining the pixel value of a single pixel is described below; the determination of the pixel values ​​of other pixels is similar. Step S622 may include: determining a conversion coefficient based on a first intensity; and determining the pixel value of the first pixel based on a preset pixel value and the conversion coefficient. Multiple pixels in a two-dimensional image include the first pixel, and the first intensity is the intensity in the point cloud data corresponding to the first pixel. This correspondence can be obtained through a point cloud mapping process (S621), where the first point in the point cloud is mapped to the first pixel, and the intensity in the attribute information of the first point is the first intensity. Step S622 may include: determining a conversion coefficient based on a first distance; and determining the pixel value of the first pixel based on a preset pixel value and the conversion coefficient. Multiple pixels in a two-dimensional image include the first pixel, and the first distance is the distance in the point cloud data corresponding to the first pixel. This correspondence can be obtained through a point cloud mapping process (S621), where the first point in the point cloud is mapped to the first pixel, and the distance in the attribute information of the first point is the first distance. The intensity or distance is transformed to obtain the conversion coefficient, and then the pixel value is obtained using the conversion coefficient. This can improve the sensitivity of the data, making the differences in intensity or distance in two-dimensional images more significant, which is more conducive to the accuracy of occlusion detection.

[0089] In some embodiments of this disclosure, intensity and distance can be used together to determine pixel values. Step S622 may include: determining a conversion coefficient based on a first intensity and a first distance; and determining the pixel value of a first pixel based on a preset pixel value and the conversion coefficient. Intensity is related to the reflectivity of an object and also to the distance to the object. Considering both intensity and distance simultaneously allows the pixel value to more accurately reflect the occlusion situation, further improving the accuracy of occlusion detection.

[0090] In some embodiments of this disclosure, intensity (or intensity and distance) can be used to convert to an echo energy coefficient, and the dynamic range of the echo energy coefficient can be adjusted to obtain a conversion coefficient. In this case, the process of determining the conversion coefficient may include: determining a first echo energy coefficient based on a first intensity or based on a first intensity and a first distance; and adjusting the dynamic range of the first echo energy coefficient to obtain a second echo energy coefficient.

[0091] Dynamic range reflects the degree of difference between data. A larger dynamic range indicates greater differences between data, making analysis and processing easier. By adjusting the dynamic range of the first echo energy coefficient, a more reasonable difference can be made between different echo energy coefficients, facilitating analysis and processing, and making the final pixel values ​​more able to distinguish the strength of echo energy. For ease of distinction, the echo energy coefficient before dynamic range adjustment is called the first echo energy coefficient, and the adjusted echo energy coefficient is called the second echo energy coefficient.

[0092] In some embodiments of this disclosure, dimensionless values ​​can be used to represent intensity. For example, the intensity range may include 0-255 (including boundary values), or 0-65535 (including boundary values). The above intensity ranges are merely examples, and the embodiments of this disclosure do not limit the intensity range.

[0093] In some embodiments of this disclosure, the first echo energy coefficient can be obtained by taking the square root of the intensity. Alternatively, the first echo energy coefficient can be obtained by taking the square root of the intensity and using distance as an adjustment coefficient. In other embodiments of this application, other processing methods can also be used to obtain the first echo energy coefficient, such as taking the square root of the intensity with a higher order, or taking the square root of the intensity with a higher order and using distance as an adjustment coefficient, etc. The number of times the intensity is taken as a square root and the number of times the distance is multiplied can be the same or different, and the number of times the distance is multiplied can be less than or equal to the number of times the intensity is taken as a square root. In some embodiments of this disclosure, the second echo energy coefficient can be obtained by performing a logarithmic operation on the first echo energy coefficient. Optionally, the first echo energy coefficient can be adjusted (e.g., by adding or subtracting a constant) before performing the logarithmic operation. Adjusting the constant can make the dynamic range adjustment more reasonable. The embodiments of this disclosure do not limit the value of the constant; it can be flexibly set as needed in application.

[0094] In some embodiments of this disclosure, equalization processing can be introduced to map pixel values ​​to a preset range, thereby reducing the span of pixel values ​​and improving data processing efficiency. In this case, the process of determining the conversion coefficients can also include: equalizing the second echo energy coefficients to obtain the conversion coefficients. The embodiments of this disclosure do not limit the method of equalization processing; the method can be selected according to the distribution of the data. For example, when the data values ​​are relatively concentrated, they can be processed using linear functions, such as min-max normalization. Conversely, when the data values ​​are highly differentiated, with some values ​​being large and others small, they can be processed using nonlinear functions. Nonlinear functions include, but are not limited to, sigmoid functions, hyperbolic tangent (tanh) functions, or rectified linear unit (ReLU) functions, or variations of these activation functions.

[0095] In the above embodiments, the equalized second echo energy coefficient is used as the conversion coefficient, and the equalized conversion coefficient can be within a preset range. For example, the equalized conversion coefficient can be within the range of 0-1 (including boundary values). This disclosure does not limit this range; in other embodiments, other ranges greater than 1 can be selected, and the range interval can be greater than 1. In some embodiments of this disclosure, the second echo energy coefficient can also be used as the conversion coefficient.

[0096] Pixel values ​​can be grayscale or color values. In grayscale images, pixel values ​​are grayscale values, ranging from 0 to 255 (inclusive). In color images, pixel values ​​can be values ​​of image components. For example, the RGB color space includes three image components (or image channels): R, G, and B, representing red, green, and blue colors respectively. In the RGB color space, a pixel's value can include the values ​​of the R, G, and B components; changing the values ​​of these components will result in different colors in the image. Similarly, the YUV color space includes three image components: Y, U, and V, where Y represents brightness, U represents hue, and V represents saturation. In the YUV color space, a pixel's value can include the values ​​of the Y, U, and V components; changing these components will result in different colors in the image. Component values ​​from different color spaces can be converted; for example, RGB components can be converted to YUV components.

[0097] In this embodiment of the disclosure, a pixel value may include the values ​​of multiple image components, such as R, G, and B components, or Y, U, and V components; or it may include the value of a single image component, such as a grayscale value. Grayscale values ​​can be used to represent the degree of brightness or darkness. This embodiment of the disclosure does not limit the pixel value or the color space.

[0098] In some embodiments of this disclosure, the preset pixel value can be a grayscale value or a color value. Optionally, the grayscale value or color value can be the maximum value. The pixel value of a pixel can be obtained by multiplying the conversion coefficient by the preset pixel value and rounding it. Assuming the conversion coefficient is N, taking a grayscale image as an example, the pixel value is a grayscale value, ranging from 0 to 255 (inclusive of boundary values). The preset pixel value can be 255, and the pixel value of the converted pixel is int(255*N), where int() represents the rounding operation, and can be rounding up or down. Assuming the conversion coefficients include N1, N2, and N3; N1, N2, and N3 can be all the same, all different, or partially the same or different. Taking an RGB image as an example, its image components include three components: R, G, and B, each ranging from 0 to 255 (inclusive of boundary values). A preset pixel value of 255 can be selected. The target values ​​for the three components of the converted pixel are 255*N1, 255*N2, and 255*N3. Performing the above rounding operation on these three target values ​​yields the pixel values ​​corresponding to the R, G, and B components. Choosing a grayscale image in occlusion detection reduces computational load, lowers the complexity of occlusion detection, and improves its efficiency.

[0099] Point cloud data can be converted into image data through point cloud mapping and point cloud pixelation. Feature extraction from the image data allows for the detection of viewport occlusion based on the extracted feature information. The following provides an illustrative description of feature extraction and occlusion determination.

[0100] The feature information includes one or more of single-frame features or inter-frame features. For example, the feature information includes one or more of image information entropy, image frequency domain energy, image edge information, and image correlation coefficient. Single-frame features include, for example, image information entropy, image frequency domain energy, or image edge information. Inter-frame features include, for example, image correlation coefficient.

[0101] Image information entropy represents the uncertainty of pixel values ​​in an image and can be used to measure the richness of image information. Higher image information entropy indicates greater uncertainty in pixel values, or richer texture information. When the viewport of an optical detection device is occluded, the texture information in the image decreases, and the image information entropy decreases compared to the unoccluded case. The occlusion state of the viewport can be determined based on image information entropy. Image information entropy can be determined based on the pixel values ​​of the pixels in the image and the frequency of those pixel values ​​in the image. Image information entropy includes, for example, full-frame information entropy or average information entropy. Full-frame information entropy refers to the information entropy of the entire image, such as the image converted from a frame of point cloud data. Average information entropy refers to the average of the information entropy of multiple sub-regions of the entire image; for example, dividing the entire image into multiple sub-regions and using the average of the information entropy of these sub-regions as the average information entropy of the entire image. When using average information entropy, the difference in information entropy between the image converted from point cloud data with occlusion and the image converted from point cloud data without occlusion is more significant, allowing for a more accurate determination of the occlusion status of the viewport.

[0102] Image frequency domain energy can be used to measure the texture or detail information of an image. Low-frequency energy reflects the overall outline or color information of the image, while high-frequency energy reflects its structural information. When the viewport is unobstructed, the image contains more high-frequency energy; when the viewport is obstructed, the high-frequency energy decreases. The occlusion state of the viewport can be determined based on the image frequency domain energy. Pixel values ​​in an image can be converted from the spatial domain to the frequency domain using Fourier transform or wavelet transform.

[0103] Image edge information can be used to represent edges or boundaries in an image, corresponding to changes in the contours or structures of objects. In image processing, edge information is commonly used for tasks such as object detection, segmentation, and recognition. When the viewport is unobstructed, the image contains more edge information; when the viewport is occluded, the edges of objects are harder to distinguish, and the image's edge information decreases. The occlusion state of a viewport can be determined based on image edge information. Edge information can be obtained through edge detection algorithms, such as performing convolution operations on the image using convolution operators (Robert operator, Prewitt operator, or Sobel operator, etc.), or detecting edges using edge detectors.

[0104] Image correlation can be characterized by the image correlation coefficient, which represents the similarity between different images. When a viewport is occluded, the position of the occluder in the image remains essentially unchanged, and the image correlation coefficient can be used to determine the occlusion status. The correlation between two frames of images with occlusion is higher than that between frames without occlusion. Image correlation coefficients may include one or more of the following: pixel standard deviation, pixel mean, and structural similarity. Pixel standard deviation or pixel mean is a measure of the pixel intensity distribution of an image, used to measure contrast or brightness variations, and can be determined based on pixel values. A larger pixel standard deviation or pixel mean indicates stronger contrast or richer brightness variations. Structural similarity measures the similarity between different images, considering brightness, contrast, and line structure information, and is more accurate in determining the similarity between different images.

[0105] In some embodiments of this disclosure, single-frame features and inter-frame features are selected to determine the occlusion state of the window, which can increase the accuracy of occlusion detection. Single-frame features include, for example, image information entropy, and inter-frame features include, for example, image correlation coefficients. Image information entropy and image correlation coefficients show a more obvious trend in image changes when occlusion exists. By comprehensively considering image information entropy and image correlation coefficients for occlusion detection, higher accuracy is achieved. For example, the correlation between two frames of images is higher when the window is occluded than when the window is not occluded; the average information entropy of the image is lower when the window is occluded than when the window is not occluded. The two frames of images can be two consecutive frames in an image sequence, i.e., two frames of point cloud data corresponding to consecutive scanning cycles of the optical detection device; or they can be two non-consecutive frames, with the time difference between the corresponding scanning cycles being less than or equal to a preset time. In other embodiments, image information entropy or image correlation coefficients can be selected for occlusion detection.

[0106] In the feature extraction process, a sliding window approach can be used. This embodiment does not limit the size of the sliding window. The size of the sliding window can be an empirical value, determined based on the size of the light spot. The sliding window is moved within the image range, and feature values ​​within the window are extracted. When the feature values ​​meet preset conditions, it is determined that the area where the sliding window is located is occluded. The sliding window is moved until it covers the entire image, and the areas where occluded sliding windows are located are merged to obtain the occluded area of ​​the viewport. Taking feature values ​​including image information entropy and image correlation coefficient as an example, when the image information entropy within the sliding window is less than or equal to a first threshold, and the image correlation coefficient is greater than or equal to a second threshold, it is determined that the area where the sliding window is located is occluded; the areas of at least one occluded sliding window are merged to obtain the occluded area of ​​the viewport.

[0107] Any of the methods in the above embodiments can be applied to the processing of a single frame of point cloud data, or to the processing of multiple frames of point cloud data. The multiple frames of point cloud data can be continuous or non-continuous. When applied to the processing of multiple frames of point cloud data, the feature information includes feature information of the multi-frame image data obtained by converting point cloud data from multiple scanning cycles; or, it includes feature information of the fused image obtained after image fusion processing. The feature information of the multi-frame image data includes, for example, single-frame features and / or inter-frame features of the multi-frame images, where the number of inter-frame features is less than or equal to the number of frames in the image.

[0108] The following describes an example of an occlusion detection method based on multiple frames of images provided in this disclosure, with reference to the accompanying drawings. Please refer to... Figure 9 The diagram illustrates a flowchart of another occlusion detection method provided in some embodiments of this disclosure.

[0109] Step S610 may include: acquiring point cloud data from multiple scan cycles, acquiring multiple point cloud frames, for example... Figure 9 The point cloud frames shown are t0-tn, where n is a positive integer greater than 1. Step S620 includes: converting point cloud data from multiple scan cycles into multi-frame image data, and acquiring multiple image frames, for example... Figure 9 The images shown are 1 to n. Step S630 may include: extracting features from the multi-frame image data to obtain feature information of the multi-frame images. The content of the feature information is the same as described above.

[0110] like Figure 9 As shown, assuming n point cloud frames are acquired, the point cloud data corresponding to the n point cloud frames is converted into image data, resulting in n image frames. Feature extraction is performed on the n image frames, for example, extracting n single-frame features and n-1 inter-frame features. Figure 9 For example, Figure 9 The inter-frame features in this example take the features of the current frame image relative to the previous frame image; in some other embodiments, inter-frame features of non-contiguous frames can also be selected for implementation. For example, the single-frame feature is the average information entropy; the inter-frame feature is the image correlation coefficient. When determining the occlusion state based on feature information, it can be determined whether there are m frames in n frames that meet the condition that m≤n. This condition is, for example, that the single-frame feature is less than or equal to a first threshold, and the inter-frame feature is greater than or equal to a second threshold. When the condition is met, it is determined that the window is occluded. The m frames can be consecutive m frames or non-consecutive m frames. Optionally, the sliding window method described in the above embodiments can be used to determine the occlusion area of ​​the window; this process includes: using the sliding window to determine the occluded sliding window area; merging the occluded sliding window areas to obtain the occlusion area of ​​the window.

[0111] The above description uses a single-frame feature (e.g., image information entropy) as an example; the judgment of other single-frame features can also be done using a similar method. Alternatively, when extracting multiple feature information, such as edge information and image information entropy, the above conditions can include edge information being less than or equal to an edge threshold, and image information entropy being less than or equal to an information entropy threshold. Joint judgment using multiple feature information is beneficial for improving the accuracy of occlusion detection. When fusing multiple frames of images into a fused image, the feature extraction method is similar to that of single-frame images, and the occlusion detection method is also similar.

[0112] In some embodiments of this disclosure, the position information of points in the point cloud can be corrected before point cloud mapping to improve the accuracy of occlusion detection. For example, a position correction process can be included before converting the point cloud data into image data (S620). Step S621 may include: correcting the position information of multiple points in the point cloud to obtain corrected position information; and determining the two-dimensional position of multiple pixels based on the corrected position information. When the position information of multiple points includes the orientation information of multiple points, correcting the position information includes correcting the first azimuth angle information and the second azimuth angle information. When the position information of multiple points includes the three-dimensional position coordinates of multiple points, correcting the position information includes correcting the three-dimensional position coordinates. In some embodiments, the optical detection device may include multiple emitting areas, and during the correction process, the position information of multiple points is corrected based on the positions of the multiple emitting areas.

[0113] For example, please refer to Figure 10 This illustrates a schematic diagram of the arrangement of light-emitting areas provided in some embodiments of this disclosure. For example... Figure 10 As shown, the squares represent luminescent areas. An optical detection device can include multiple luminescent areas arranged in an array. However, during the arrangement process, space limitations on the chip or board may cause different luminescent areas to be misaligned in their arrangement direction (e.g., horizontal or vertical). For example, there may be a horizontal angular difference between different luminescent columns. This horizontal angular difference is reflected in the point cloud data and subsequently in the converted image data. In some implementations, the positional information of multiple points can be corrected, thereby improving image quality and further enhancing the accuracy of occlusion detection. In some implementations, positional information correction is not required; occlusion detection has a certain tolerance for misalignment between sub-regions in the image, so occlusion detection can be achieved without positional information correction.

[0114] In the implementation of the light-emitting areas, each light-emitting area can correspond to one laser, and the driving circuit controls the light-emitting area corresponding to that laser by driving the laser. Alternatively, each light-emitting area can correspond to multiple lasers (called a laser group), and the driving circuit controls the light-emitting area corresponding to that laser group by driving the laser group. Or, one laser can correspond to multiple light-emitting areas. For example, the laser is provided with multiple light-emitting points (also called light-emitting apertures), and the light-emitting points of different light-emitting areas can use different pads. The driving circuit can drive the light-emitting points through the pads to control the light-emitting area corresponding to the light-emitting point. The laser can be, for example, a semiconductor laser, a fiber laser, or other types of lasers. Semiconductor lasers include, for example, laser emitting circuits, vertical cavity surface emitting lasers (VCSELs), edge emitting lasers (EELs), distributed feedback lasers (DFBs), or similar devices. The above are merely examples, and the embodiments disclosed herein do not limit the type of laser.

[0115] In some embodiments of this disclosure, the result of the first occlusion detection can be combined with the detection results of other occlusion detection methods (hereinafter referred to as the result of the second occlusion detection). Combining the detection results obtained from multiple different types of occlusion detection methods for occlusion determination can improve the accuracy of occlusion detection. Please refer to... Figure 11 This illustrates a flowchart of yet another occlusion detection method provided in some embodiments of this disclosure. Compared to... Figure 6 The occlusion detection method shown may further include:

[0116] S111: Obtain first information, which is used to indicate the second occlusion state of the window;

[0117] S112: Determine the occlusion status of the window based on the first occlusion status and the second occlusion status.

[0118] In step S112 above, the result of the first occlusion detection can be used to verify the result of the second occlusion detection, or the result of the second occlusion detection can be used to verify the result of the first occlusion detection; or the results of different occlusion detections can be combined to jointly determine the occlusion situation. For example, the result of the first occlusion detection can be used to verify the result of the second occlusion detection. When the second occlusion status shows that the window is occluded, the first occlusion detection is activated. When the result of the first occlusion detection (i.e., the first occlusion status) shows that the window is occluded, it is determined that the window is occluded. When the result of the first occlusion detection (i.e., the first occlusion status) shows that the window is not occluded, it is determined that the window is not occluded or a notification message is sent to the user. This notification message is used to notify the user to check the occlusion situation of the optical detection device. Similarly, the result of the second occlusion detection can be used to verify the result of the first occlusion detection. When the first occlusion state indicates that the window is occluded, the second occlusion detection is initiated. When the result of the second occlusion detection (i.e., the second occlusion state) indicates that the window is occluded, it is determined that the window is occluded. When the result of the second occlusion detection (i.e., the second occlusion state) indicates that the window is not occluded, it is determined that the window is not occluded or a notification message is sent to the user. This notification message is used to notify the user to check the occlusion status of the optical detection device. This disclosure does not limit the form of the notification message. The notification message can be sent to the user through the human-machine interface of the terminal device where the optical detection device is located. For example, if the terminal device is a vehicle, the notification message can be displayed on the vehicle's in-vehicle screen (including the instrument panel screen, entertainment screen, or screen with both instrument and entertainment functions), played through the speaker, or sent to the user's portable terminal (e.g., mobile phone, or watch).

[0119] In some embodiments of this disclosure, the second occlusion detection can utilize a detection laser and / or detector to obtain the result of the second occlusion detection (or the second occlusion state). For example, a laser and / or detector specifically designed for occlusion detection can be incorporated within the optical detection device. Please refer to... Figure 12 This illustrates a schematic diagram of an occlusion detection structure provided in some embodiments of this disclosure. For example... Figure 12 As shown, in addition to a laser, the optical detection device also includes a detection laser. This detection laser emits detection light in a specific direction. If there is an obstruction on the window, the optical path of the detection light is blocked, generating reflected light at the obstruction point. This reflected light can be received by the detector. Whether or not the reflected light is received when the detection light is emitted indicates whether an obstruction exists in the window. Using the first obstruction detection method provided in the above embodiment instead of this detection scheme can reduce the need for additional components in the optical detection device, thus reducing its complexity and size. Combining the first obstruction detection method and the second obstruction detection method provided in the above embodiment allows for the integration of the detection results from both methods, improving the accuracy of obstruction detection.

[0120] In some embodiments of this disclosure, the second occlusion detection can utilize stray light to obtain the result of the second occlusion detection (or the second occlusion state). Considering that the window itself also reflects some of the emitted laser light, forming internal stray light, the time-of-flight (ToF) of the stray light is typically less than the time-of-flight of the echo reflected after the laser encounters an object in the environment. After the stray light is received by the detector, a pulse corresponding to the stray light can be represented on the detector's waveform. For example, as... Figure 13 The diagram illustrates a waveform of a detector provided in some embodiments of this disclosure. If the window is obstructed, it causes waveform changes such as increased stray light pulse amplitude and pulse width broadening. Therefore, the presence of obstruction can be determined by observing the changes in the stray light pulse waveform. Combining the first obstruction detection and the second obstruction detection provided in the above embodiments, the accuracy of obstruction detection can be improved by combining the detection results of the two detections.

[0121] This disclosure also provides an occlusion detection apparatus, including units or means for performing the steps of any of the above occlusion detection methods. For example, please refer to... Figure 14 This illustration shows a structural schematic diagram of an occlusion detection device provided in some embodiments of the present disclosure. For example... Figure 14 As shown, the occlusion detection device 140 includes: an acquisition unit 141 for acquiring point cloud data; a conversion unit 142 for converting the point cloud data into image data; a feature extraction unit 143 for extracting features from the image data to obtain feature information; and an occlusion determination unit 144 for determining a first occlusion state of the window based on the feature information; or, obtaining a first occlusion detection result based on the feature information.

[0122] It should be understood that the above division of units is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the above units can be implemented by a processor calling software; for example, the system includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit. The processor can be, for example, a general-purpose processor, such as a CPU, and the memory can be internal or external to the device. Alternatively, the above units can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuit, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented through the design of the logical relationships between the internal components. In another implementation, the hardware circuit can be implemented using a PLD, which can include a large number of logic gates. The logical relationships between the logic gates are configured through a configuration file, thereby implementing the functions of some or all of the above units. All units of the above system can be implemented entirely through processor calling programs, or entirely through hardware circuits, or partially through processor calling programs with the remaining parts implemented through hardware circuits.

[0123] In this embodiment, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a CPU, MCU, graphics processing unit (GPU), or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits, where these relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as a field-programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), or deep learning processing unit (DPU).

[0124] As can be seen, each unit in the above system can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, MCU, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0125] Furthermore, the units in the above systems can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). This SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the above apparatus.

[0126] For example, please refer to Figure 15 This is a schematic diagram of another occlusion detection device provided in an embodiment of this disclosure. Figure 15 As shown, the occlusion detection device 150 includes a processor 151 for calling instructions stored in the memory 152. When the instructions are called by the processor 151, the processor 151 executes any of the occlusion detection methods in the above embodiments.

[0127] Furthermore, embodiments of this disclosure also provide a computer-readable storage medium, including instructions stored thereon, which, when invoked by a processor, execute any of the occlusion detection methods described in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk storage medium, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The above-mentioned computer-readable storage medium may, for example, include a non-volatile computer storage medium. This disclosure also provides a non-volatile computer storage medium storing instructions that, when invoked by a processor, cause the processor to execute any of the occlusion detection methods described in the above embodiments.

[0128] This disclosure also provides a computer program (or computer program product) including instructions that, when invoked by a processor, execute any of the occlusion detection methods described in the above embodiments.

[0129] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail or in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, the above embodiments can be freely combined as needed.

Claims

1. An occlusion detection method for performing a first occlusion detection on the viewing window of an optical detection device, characterized in that, The occlusion detection method includes: Acquire point cloud data; Convert the point cloud data into image data; Feature extraction is performed on the image data to obtain feature information; The first occlusion state of the window is determined based on the feature information.

2. The occlusion detection method according to claim 1, characterized in that, The point cloud data includes the location and attribute information of multiple points, and the image data includes the two-dimensional location and pixel value of multiple pixels. The step of converting the point cloud data into the image data includes: The two-dimensional positions of the multiple pixels are determined based on the position information of multiple points in the point cloud data. Based on the attribute information of multiple points in the point cloud data, the pixel values ​​of the multiple pixels are determined, wherein the attribute information includes one or both of intensity and distance.

3. The occlusion detection method according to claim 2, characterized in that, The location information of the multiple points includes direction information, which includes first azimuth information and second azimuth information; or... The location information of the multiple points includes three-dimensional location coordinates.

4. The occlusion detection method according to claim 2 or 3, characterized in that, Determining the pixel values ​​of the multiple pixels based on the attribute information of multiple points in the point cloud data includes: Based on the first intensity, determine the conversion coefficient; Based on the preset pixel value and the conversion coefficient, the pixel value of the first pixel is determined, wherein the plurality of pixels includes the first pixel, and the first intensity is the intensity in the point cloud data corresponding to the first pixel.

5. The occlusion detection method according to claim 4, characterized in that, Determining the conversion coefficient based on the first intensity includes: Based on the first intensity and the first distance, the conversion coefficient is determined, where the first distance is the distance in the point cloud data corresponding to the first pixel.

6. The occlusion detection method according to claim 4 or 5, characterized in that, Determining the conversion coefficients includes: The first echo energy coefficient is determined based on the first intensity or based on the first intensity and the first distance; Adjust the dynamic range of the first echo energy coefficient to obtain the second echo energy coefficient; The second echo energy coefficient is equalized to obtain the conversion coefficient.

7. The occlusion detection method according to any one of claims 2-6, characterized in that, Determining the two-dimensional position of the multiple pixels based on the position information of multiple points in the point cloud data includes: The position information of the multiple points is corrected to obtain corrected position information; The two-dimensional positions of the plurality of pixels are determined based on the corrected position information.

8. The occlusion detection method according to claim 7, characterized in that, The field of view of the optical detection device includes multiple luminescent areas, and the correction of the position information of the multiple points includes: Based on the positions of the multiple luminescent areas, the position information of the multiple points is corrected.

9. The occlusion detection method according to any one of claims 1-8, characterized in that, Also includes: Obtain first information, which is used to indicate the second occlusion state of the window; The occlusion status of the viewport is determined based on the first occlusion status and the second occlusion status.

10. The occlusion detection method according to any one of claims 1-9, characterized in that, The feature information includes one or more of single-frame features and inter-frame features.

11. The occlusion detection method according to any one of claims 1-10, characterized in that, The feature information includes one or more of the following: image information entropy, image frequency domain energy, image edge information, and image correlation coefficient.

12. The occlusion detection method according to claim 10 or 11, characterized in that, Determining the first occlusion state of the viewport based on the feature information includes: The first occlusion state of the window is determined based on single-frame features and inter-frame features.

13. The occlusion detection method according to claim 12, characterized in that, The single-frame features include image information entropy, and the inter-frame features include image correlation coefficients.

14. The occlusion detection method according to any one of claims 10-13, characterized in that, The feature information includes feature information of multiple frames of image data obtained by converting the point cloud data over multiple scanning cycles.

15. An occlusion detection device for performing a first occlusion detection on the window of an optical detection device, characterized in that, The device includes: The acquisition unit is used to acquire point cloud data; A conversion unit is used to convert the point cloud data into image data; The extraction unit is used to extract features from the image data to obtain feature information; A determining unit is used to determine the first occlusion state of the window based on the feature information.

16. An occlusion detection device, characterized in that, For performing a first occlusion detection on the window of an optical detection device, the occlusion detection device includes a processor for calling instructions stored in a memory, wherein when the instructions are executed, the occlusion detection device performs the occlusion detection method as described in any one of claims 1-14.

17. An occlusion detection system, characterized in that, include: An optical detection device for acquiring sensing data, the sensing data including point cloud data or point cloud data for determining point cloud data; An occlusion detection device is communicatively connected to the optical detection device and is used to perform the occlusion detection method as described in any one of claims 1-14.

18. A terminal device, characterized in that, Including the occlusion detection system as described in claim 17.

Images