Circuit for point cloud calibration and laser radar

Abstract

The embodiment of the invention provides a circuit for point cloud calibration and a laser radar, and the circuit comprises a controller which is configured to generate a point cloud calibration instruction; a memory configured to store the echo data to be calibrated and the weight of the deep learning model; and the processor is configured to respond to the point cloud calibration instruction received from the controller, acquire the echo data to be calibrated and the weight of the deep learning model from the memory, and calibrate the point cloud through the deep learning model to obtain calibrated point cloud data.

Description

Circuits and lidar for point cloud calibration Technical Field

[0001] This disclosure relates to the field of point cloud calibration technology, and more particularly to circuits and lidar for point cloud calibration. Background Technology

[0002] LiDAR is short for Light Detection and Ranging. It is used to acquire point cloud data and generate accurate digital 3D models.

[0003] As an advanced sensor widely used in autonomous vehicles, robot navigation, geographic information systems, and environmental monitoring, the accuracy and reliability of lidar are crucial. Calibrating lidar point clouds, such as through filtering and distance linearity calibration, helps the lidar data accurately reflect environmental characteristics.

[0004] Currently, point cloud calibration is implemented in the Central Processing Unit (CPU). To meet the perception requirements of intelligent or autonomous driving, LiDAR continuously generates large amounts of point cloud data in real time. Current CPUs cannot handle the required computational load. The computational load required for point cloud calibration may need to be completed by multiple CPUs or multiple CPU cores. This requires a large circuit area, making it difficult to miniaturize the LiDAR.

[0005] The content in the background section is merely technology known to the public and does not represent existing technology in this field. Summary of the Invention

[0006] This disclosure provides a circuit and lidar for point cloud calibration, which can reduce the circuit area and improve the point cloud calibration efficiency while ensuring point cloud calibration performance.

[0007] This disclosure provides a circuit for point cloud calibration, comprising:

[0008] The controller is configured to generate point cloud calibration instructions;

[0009] A memory configured to store the echo data to be calibrated and the weights of the deep learning model; and

[0010] The processor is configured to respond to the point cloud calibration command received from the controller, retrieve the echo data to be calibrated and the weights of the deep learning model from the memory, and calibrate the point cloud using the deep learning model to obtain calibrated point cloud data.

[0011] Optionally, the processor includes a plurality of processing units configured to execute the deep learning model to calibrate the point cloud.

[0012] Optionally, the plurality of processing units are arranged in an array, wherein any processing unit is communicatively coupled to the processing units in adjacent rows and columns.

[0013] Optionally, the processing unit includes a multiply-accumulator.

[0014] Optionally, the echo dataset can be input into a deep learning network for training to obtain the weights of the deep learning model.

[0015] Optionally, the deep learning network includes at least one of the following:

[0016] Multilayer perceptron;

[0017] Convolutional neural networks; or

[0018] Recurrent neural networks.

[0019] Optionally, the point cloud calibration command is configured to characterize at least one of the following:

[0020] Filtering; or

[0021] Echo time or time-of-flight calibration.

[0022] Optionally, the memory includes static random access memory.

[0023] Optionally, the circuit further includes:

[0024] An analog-to-digital converter (ADC) is communicatively coupled to the memory. The ADC is configured to perform sampling and quantization operations on the echo signal received by the circuit at a preset sampling frequency to obtain the echo data to be calibrated and store it in the memory.

[0025] This disclosure also provides a lidar, including:

[0026] A laser, configured to output a detection signal to the outside;

[0027] The detector is configured to receive the echo signal returned after the detection signal is reflected by an object;

[0028] The controller is configured to control the laser to output a detection signal and the detector to receive an echo signal, and to generate a point cloud calibration command.

[0029] A memory configured to store the echo data to be calibrated and the weights of the deep learning model; and

[0030] The processor is configured to respond to the point cloud calibration command by retrieving the echo data to be calibrated and the weights of the deep learning model from the memory, and calling the deep learning model to calibrate the point cloud to obtain calibrated point cloud data.

[0031] Optionally, the processor includes a plurality of processing units configured to execute the deep learning model to calibrate the point cloud.

[0032] Optionally, the plurality of processing units are arranged in an array, wherein any processing unit is communicatively coupled to the processing units in adjacent rows and columns.

[0033] Optionally, the processing unit includes a multiply-accumulator.

[0034] Optionally, the echo dataset can be input into a deep learning network for training to obtain the weights of the deep learning model.

[0035] The circuit for point cloud calibration according to embodiments of this disclosure allows the processor to retrieve the echo data to be calibrated and the weights of the deep learning model from the memory in response to a point cloud calibration command received from the controller. The processor then calibrates the point cloud using the deep learning model to obtain calibrated point cloud data. In this circuit, the processor and the controller can operate collaboratively. The controller can generate point cloud calibration commands. The processor can calibrate the point cloud using the deep learning model in response to these commands. This reduces circuit area while ensuring the time and computational requirements for generating point cloud data for the LiDAR.

[0036] In an optional embodiment, the processor includes multiple processing units. These multiple processing units can execute the deep learning model in parallel to calibrate the point cloud. This improves the efficiency of point cloud calibration.

[0037] In an optional embodiment, the plurality of processing units are arranged in an array. Each processing unit is communicatively coupled to the processing units in adjacent rows and columns. This enables rapid data sharing between processing units and improves data processing efficiency.

[0038] In an optional embodiment, the processing unit includes a multiply-accumulator. The multiply-accumulator is used for multiplication-addition operations, which can improve the processing efficiency of matrix multiplication, convolution, or circular convolution in deep learning models.

[0039] In an optional embodiment, the echo dataset is input into a deep learning network for training to obtain the weights of the deep learning model. This can improve the point cloud calibration accuracy of the circuit. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the description of the embodiments of this disclosure or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 shows an example block diagram of a lidar consistent with some embodiments of this disclosure.

[0042] Figure 2 shows a schematic diagram of a circuit for point cloud calibration that is consistent with some embodiments of this disclosure.

[0043] Figure 3 shows a schematic diagram of the structure of a processor consistent with some embodiments of this disclosure.

[0044] Figure 4 shows a schematic diagram of a lidar structure consistent with some embodiments of this disclosure. Detailed Implementation

[0045] LiDAR (Light Detection and Ranging) uses laser light as a medium for object detection and has found applications in many fields. For example, LiDAR can be used in autonomous driving, industrial manufacturing, drones, robot recognition, geographic mapping, and environmental monitoring. Autonomous driving, also known as automated driving or assisted driving, includes any level of automated driving, such as L1-L5. In applications, LiDAR can be mounted on vehicles to provide them with perception data, such as point cloud data, enabling the vehicles to perform one or more functions such as analysis, decision-making, or control. Vehicles include, but are not limited to, vehicles, manufacturing terminals, ships, aircraft (such as flying vehicles or drones), robots (such as industrial robots or home robots), or surveying equipment.

[0046] LiDAR can include, but is not limited to, mechanical LiDAR, semi-solid-state LiDAR, or solid-state LiDAR. Semi-solid-state LiDAR can include, but is not limited to, microelectromechanical system (MEMS) LiDAR, rotating mirror LiDAR, tilting mirror LiDAR, or prism LiDAR; solid-state LiDAR can include, but is not limited to, optical phase array (OPA) LiDAR, or flash LiDAR. When a vehicle is equipped with multiple LiDARs, the types of LiDARs can be the same or different.

[0047] Figure 1 shows an example block diagram of a lidar consistent with some embodiments of the present disclosure. Referring to Figure 1, lidar 100 may include a laser emitting circuit 102, a laser receiving circuit 104, an optical system 106, and a control and processing system 108.

[0048] In some embodiments, the laser emitting circuit 102 can emit a laser. The laser is emitted after the optical path is adjusted by the optical system 106. When the emitted laser encounters an object, it is reflected by the object. The light reflected back to the lidar 100 can be called an echo. The echo is then directed to the laser receiving circuit 104 after the optical path is adjusted by the optical system 106. The laser receiving circuit 104 can receive the echo and convert it into an electrical signal. The echo data obtained based on the electrical signal is provided to the control and processing system 108. The control and processing system 108 can process the echo data to obtain sensing data (e.g., point cloud data). The control and processing system 108 can also send the sensing data to a vehicle. The vehicle can use the sensing data to perform one or more functions such as analysis, decision-making, or control.

[0049] In some embodiments, the laser emitting circuit 102 includes a driving circuit 110 and a laser 112. The laser 112 emits laser light under the drive of the driving circuit 110. The laser light exits through an optical system 106. For example, the laser 112 includes a semiconductor laser, a fiber laser, or other types of lasers. For example, a semiconductor laser includes a laser emitting circuit, a vertical cavity surface emitting laser (VCSEL), an edge emitting laser (EEL), a distributed feedback laser (DFB), or similar devices. The above are merely examples, and the embodiments disclosed herein do not limit the type of laser.

[0050] In some embodiments, the laser receiving circuit 104 includes a detector 114. The optical system 106 can focus the echo reflected from an object onto the photosensitive surface of the detector 114. The detector 114 can convert the optical signal into an electrical signal using the photoelectric effect. The laser receiving circuit 104 may also include a gating circuit 118. The gating circuit 118 can select some or all of the detectors 114. The selected detectors 114 are in a state that can respond to optical signals and can convert the echo into an electrical signal. For example, the detector 114 may include: a photodetector circuit, a PIN photodiode (PINPD), an avalanche photodiode (APD), a single photon avalanche diode (SPAD), a silicon photomultiplier (SiPM), or similar devices. The above are merely examples, and the embodiments disclosed herein do not limit the type of detector.

[0051] In some embodiments, the laser receiving circuit 104 may further include a preprocessing circuit 116. For example, preprocessing includes one or more of amplification or sampling processes. Preprocessing can be implemented by the preprocessing circuit 116. For example, the preprocessing circuit 116 may include one or more of an amplification circuit, a filtering circuit, or a sampling circuit. For example, the amplification circuit includes an amplifier. The amplifier can amplify the electrical signal converted by the detector. The sampling circuit can sample the electrical signal output by the detector to obtain echo data.

[0052] The optical system 106 may include one or more sets of optical components, each set of optical components including at least one optical element. In some embodiments, the optical system 106 includes a emitting optical element and a receiving optical element. The emitting optical element, along the emission path of the laser, can shape the laser emitted by the laser 112 and adjust the laser's exit path. The receiving optical element, along the receiving path of the laser, can collect the echo reflected back from the object and converge the echo onto the photosensitive surface of the detector 114. For example, the emitting optical element includes one or more optical elements such as a emitting lens, a mirror, a homogenizer, or a beam splitter. For example, the receiving optical element includes one or more optical elements such as a receiving lens, a mirror, a filter, or a beam splitter (or beam splitter). The emitting and receiving optical elements can be independent, partially multiplexed, or fully multiplexed. For example, a lidar may include independent emitting and receiving optical elements, such as independent emitting and receiving lenses. For example, a lidar may include optical elements shared by the emitting and receiving optical paths, such as a beam splitter. A beam splitter can be used to separate the transmitting optical path and the receiving optical path. For example, a lidar system may include a common lens for shaping the coaxial beam in both the transmitting and receiving optical paths.

[0053] In some embodiments, the control and processing system 108 can process the echo data to obtain sensing data. The control and processing system 108 can also send control signals to the drive circuit 110 to control the drive circuit 110 to drive the laser 112 to emit laser light. The control and processing system 108 can also send control signals to the gating circuit 118 to control the gating circuit 118 to select some or all of the detectors 114. For example, the control and processing system 108 may include one or more processors. Processors include, but are not limited to, application-specific integrated circuits (ASICs), hardware circuits implemented with programmable logic devices (PLDs), microcontroller units (MCUs), microprocessor units (MPUs), digital signal processors (DSPs), or central processing units (CPUs). For example, hardware circuits implemented with PLDs may include field-programmable gate arrays (FPGAs). When the control and processing system 108 includes multiple processors, the types of processors may be the same or different. For example, control and processing system 108 may include an MCU and an FPGA. Control and processing system 108 may include an MCU, an FPGA, and a CPU. Control and processing system 108 may include an MCU, a DSP, and an FPGA. Alternatively, control and processing system 108 may include a CPU and an FPGA, and so on. When control and processing system 108 includes multiple processors, these processors may be configured separately, partially integrated, or fully integrated. For example, control and processing system 108 may be implemented as a system-on-chip (SOC) or an ASIC.

[0054] Optionally, the lidar 100 may further include a scanning system 120. Taking a mechanical lidar or a semi-solid-state lidar as an example, the scanning system 120 may include a scanner and a driver. The driver can drive the scanner to rotate, enabling the laser to scan one or all of the vertical or horizontal field of view. For example, the laser is emitted through the scanner, and the rotation of the scanner can change the emission path of the laser. Alternatively, the laser echo can be incident on the scanner and guided to the light receiving path. This disclosure does not limit the type of scanner. For example, the scanner may include, but is not limited to, a rotating mirror, a tilting mirror, a galvanometer, or other components that can direct the laser to different directions in the environment. Furthermore, the scanning system 120 may include a rotating platform. One or more of a laser emitting circuit, a laser receiving circuit, or an optical system may be mounted on the rotating platform, and as the platform rotates, scanning one or all of the vertical or horizontal field of view can be achieved. When the lidar 100 includes the scanning system 120, the control and processing system 108 can also control the movement of the scanning system 120.

[0055] In some embodiments, the lidar 100 may not include a scanning system, and may not include mechanical rotating parts. For example, the lidar 100 includes multiple lasers 112 and multiple detectors 114. The multiple lasers 112 and multiple detectors 114 are arranged in a two-dimensional array. The lasers emitted by the lasers 112 at different positions in the two-dimensional array are emitted at different angles after passing through the optical system 106, thereby realizing the two-dimensional scanning of the lidar. In actual detection, the laser receiving circuit 104 may be affected by factors such as photoelectric response and circuit wiring, causing the echo data to deviate from the true value. Crosstalk may also exist between multiple detectors. In addition, when the scanning trajectory is too long, the point cloud is prone to drift errors and decreased accuracy. As can be seen from the above, the point cloud data generated by the lidar may not accurately reflect the environmental information. To this end, the lidar can perform point cloud calibration, such as filtering and distance linearity calibration. This can improve the accuracy of the echo data.

[0056] Consistent with some embodiments of this disclosure, in order to meet the perception requirements of intelligent driving or autonomous driving, LiDAR can continuously generate a large amount of point cloud data in real time. The continuous detection of environmental information by LiDAR and the output of corresponding point cloud data can assist in the decision-making of intelligent driving or autonomous driving. After the detector receives the echo signal, preprocessing and point cloud calibration require some runtime. There is a time required for operation or computation between the detector receiving the echo signal and the LiDAR outputting the point cloud data. The decision-making of intelligent driving or autonomous driving places high demands on the real-time performance of detection, limiting the operation or computation to be completed in the shortest possible time.

[0057] Point cloud calibration can be achieved in the control and processing system 108 through filtering or distance calibration algorithms. For example, the control and processing system 108 may include a central processing unit (CPU). The CPU performs point cloud calibration on the echo data, obtaining calibrated point cloud data output. The calibrated point cloud data can accurately reflect environmental information. The computational load required for point cloud calibration may need to be completed by multiple CPUs or multiple CPU cores. This requires a large circuit area, making it difficult to reduce the size of the LiDAR.

[0058] This disclosure provides a circuit for point cloud calibration. In some embodiments, a controller and a processor can work together. The controller can generate point cloud calibration instructions. The processor can execute a deep learning model to calibrate the point cloud, obtaining calibrated point cloud data. This reduces circuit area and ensures that the LiDAR meets the time and computational requirements for generating point cloud data.

[0059] In some embodiments, the circuit can be applied to a lidar system. This can facilitate the miniaturization of lidar systems using the circuit.

[0060] The following detailed description is provided with reference to the accompanying drawings, through specific embodiments and in conjunction with some specific application scenarios.

[0061] Figure 2 shows a schematic diagram of a circuit for point cloud calibration consistent with some embodiments of this disclosure. Referring to Figure 2, in some embodiments, the circuit 200 for point cloud calibration may include a controller 202, a memory 204, and a processor 206. For example, the control and processing system 108 of Figure 1 may include circuit 200.

[0062] Controller 202 can generate point cloud calibration instructions.

[0063] The memory 204 can store the echo data to be calibrated and the weights of the deep learning model.

[0064] The processor 206 can respond to point cloud calibration commands received from the controller 202 and retrieve the echo data to be calibrated and the weights of the deep learning model from the memory 204. The processor 206 can also calibrate the point cloud using the deep learning model to obtain calibrated point cloud data.

[0065] In some embodiments, the processor 206 and the controller 202 can operate collaboratively. The controller 202 can generate point cloud calibration instructions. The processor 206 can calibrate the point cloud using a deep learning model in response to the point cloud calibration instructions. This ensures that the time and computational requirements for generating point cloud data by the LiDAR are met. Some embodiments of this disclosure do not require multiple controllers or processors, or multiple processor cores. In this case, the circuit area can be reduced.

[0066] In some embodiments, point cloud calibration instructions may characterize one or more of filtering, echo time, or time-of-flight calibration.

[0067] For example, filtering can remove noise from echo data or remove isolated points. For instance, for a point in the point cloud to be determined, processor 206 can determine the neighborhood of the point. The neighborhood can be points in the point cloud that are angularly adjacent to the point to be determined, or a set of points whose angular differences from the point to be determined are within a preset range. Processor 206 can determine the distance between the point to be determined and other points in the neighborhood. If the distance is greater than a preset threshold, processor 206 can determine that the point to be determined is an isolated point and remove it from the point cloud. Processor 206 can determine the preset range and preset threshold of the angular difference based on factors such as the angular resolution and scanning speed of the LiDAR.

[0068] The processor 206 can also determine the number of points in the neighborhood. If the number of points in the neighborhood is less than a point count threshold, the processor 206 can determine that the point to be judged is an isolated point and remove it from the point cloud. The processor 206 can determine the point count threshold based on at least one of the following factors: the angular resolution of the LiDAR, power consumption, or noise removal effect.

[0069] For example, due to hardware characteristics or environmental factors, the measurement results of lidar may contain systematic biases. By measuring a reference object at a known distance, the bias of the lidar at different distances is detected and recorded. The lidar can store the time-of-flight bias of each detection channel at different distances. In real-time detection, the processor 206 can calibrate the time of flight represented by the echo data to be calibrated based on the time-of-flight bias corresponding to the echo data to be calibrated. When laser light is reflected by objects with different reflectivities, the shape of the echo pulse may change. For example, the echo pulses of objects with different reflectivities have different pulse widths, leading-edge slopes, etc. This may cause deviations in the time of flight. The processor 206 can also calibrate the time-of-flight bias caused by reflectivity.

[0070] In some embodiments, the controller 202 may include circuitry with signal processing capabilities.

[0071] For example, controller 202 may include circuitry or devices with instruction generation capabilities. For instance, controller 202 may include circuitry or devices such as a CPU, microcontroller (MCU), microprocessor unit (MPU), or digital signal processor (DSP).

[0072] In some embodiments, the controller 202 may also be a combination of one or more devices or circuits.

[0073] For example, the controller may include a CPU. The CPU may include an arithmetic logic unit, a register unit, and a control unit. The register unit can be used to temporarily store instructions, data, and addresses. The control unit can be used to generate a point cloud calibration command when it detects the presence of echo data to be calibrated in the memory 204, or when it detects that the amount of echo data stored in the memory 204 has reached a preset threshold. The control unit can also issue control signals corresponding to the point cloud calibration command.

[0074] In some embodiments, echo data may include a digital echo signal obtained by analog-to-digital conversion of an analog echo signal, and echo information data associated with the echo signal. For example, the digital echo signal can be used to determine the time of flight, and thus the distance to an object. The digital echo signal can also be used to determine information such as the reflectivity or velocity of an object. The echo information data may include information such as the detection angle or detection channel corresponding to the digital echo signal.

[0075] In some embodiments, memory 204 may include a device or circuit with storage function.

[0076] For example, memory 204 may include read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions. Memory 204 may also include random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions. For example, memory 204 may include static random access memory (SRAM). When SRAM is powered on, the data stored inside can be constantly retained. SRAM can be used for caching to improve data access speed.

[0077] In some embodiments, processor 206 can implement certain functions through the logical relationships of hardware circuits. These logical relationships can be fixed or reconfigurable. Exemplarily, processor 206 may include hardware circuits implemented using a programmable logic device (PLD). For example, processor 206 may include a field-programmable gate array (FPGA). In reconfigurable hardware circuits, processor 206 loads a configuration document to configure the hardware circuits. For example, processor 206 loads a point cloud calibration document to perform point cloud calibration.

[0078] In some embodiments, processor 206 may also include hardware circuitry designed for artificial intelligence. Such hardware circuitry includes application-specific integrated circuits (ASICs). Exemplarily, processor 206 may include a combination of one or more devices such as a neural network processing unit (NPU), a tensor processing unit (TPU), or a deep learning processing unit (DPU).

[0079] In some embodiments, the processor 206 may also include a combination of one or more devices or circuits. Point cloud calibration of this disclosure can be implemented by configuring any combination of one or more of these devices or circuits.

[0080] In some embodiments, any two or three of the controller 202, memory 204, and processor 206 may be communicatively coupled. Exemplarily, any two or three of the controller 202, memory 204, and processor 206 may communicate data via one or more methods such as electrical connection, magnetic connection, optical connection, wireless connection, and wired connection. This disclosure does not limit the connection method or data communication method between the controller 202, memory 204, and processor 206. For example, any two or three of the controller 202, memory 204, and processor 206 may be connected via a bus. This can improve data communication speed.

[0081] In some embodiments, processor 206 may include multiple processing units. These multiple processing units can execute deep learning models in parallel to calibrate point clouds. This improves the efficiency of point cloud calibration.

[0082] In some embodiments, the echo dataset can be input into the deep learning network for training to obtain the weights of the deep learning model. Training the deep learning network with an echo dataset containing a large amount of echo data allows the weights of the deep learning model to more realistically and accurately reflect the biases in the echo data.

[0083] Echo datasets can include echo data from multiple channels. For example, echo data can include digital echo signals and echo information data. Information such as the detection angle or detection channel corresponding to the digital echo signal can be determined based on the echo information data. During training, the echo dataset can be input into a deep learning network to train weights for point cloud calibration.

[0084] In some embodiments, a deep learning model is trained after the LiDAR is assembled. The weights of the trained deep learning network are stored in a memory. Exemplarily, a deep learning model is trained separately for each LiDAR. The echo dataset used to train the deep learning model can be the same or different for different LiDARs.

[0085] In some embodiments, new echo data can be collected during the operation of the LiDAR. The echo dataset used for training the deep learning model can be updated. After the LiDAR has been running for a period of time, the deep learning model can be retrained using the updated echo dataset to obtain updated weights for the deep learning network. This allows for flexible responses to emerging data processing needs. The point cloud calibration function can be adjusted at the software level without requiring hardware changes.

[0086] In some embodiments, a deep learning network may include a multi-layer perceptron (MLP). An MLP is a feedforward artificial neural network. An MLP comprises multiple neurons. For example, an MLP may include an input layer, one or more hidden layers, and an output layer. Each node in each layer can connect to each node in adjacent layers, thus an MLP is a fully connected neural network. The main operation in a fully connected neural network is matrix multiplication. The relationship between each layer and its adjacent layers in an MLP can be fitted using the trained layer number, weights, and activation functions to realistically reflect the deviation, either linearly or nonlinearly. For example, each node in an MLP is implemented through a processing unit.

[0087] In some embodiments, a deep learning network may include convolutional neural networks (CNNs). A CNN is a type of feedforward neural network that includes convolutional operations and has a deep structure. For example, a CNN may include an input layer, hidden layers, and an output layer.

[0088] The data input layer can process multidimensional data. In some embodiments, the data input layer can standardize the input echo data. For example, the input data layer can normalize the input echo data to be calibrated. Standardization of input features can improve the learning efficiency and performance of convolutional neural networks.

[0089] Hidden layers can include multi-layer structures such as convolutional layers, pooling layers, and fully connected layers.

[0090] Convolutional layers extract features from input echo data. A convolutional layer can contain multiple convolutional kernels. Each element of a convolutional kernel corresponds to a weight and a bias, and this element can be called a neuron. Each neuron in a convolutional layer is connected to multiple neurons in a nearby region from the previous layer. The size of this region depends on the size of the convolutional kernel and can be called the "receptive field." Each convolutional kernel can be viewed as a filter. The size of the filter can be obtained through training. When working, the convolutional kernel scans the input feature data with a certain stride, performing matrix multiplication and summation on the input feature data within the receptive field, and then adding the bias.

[0091] After feature extraction in the convolutional layer, the output feature map can be passed to the pooling layer for feature selection and information filtering. The fully connected layer can perform non-linear combination of the extracted features and output the result.

[0092] In some embodiments, the deep learning network may include a recurrent neural network (RNN). An RNN is a neural network architecture used to process sequential data. By introducing recurrent connections, RNNs enable the network to capture temporal dependencies and contextual information within the sequence. For example, the echo data to be calibrated may include a temporally continuous sequence of echo data. Performing point cloud calibration based on echo data corresponding to consecutive point cloud frames can improve calibration accuracy.

[0093] In some embodiments, processor 206 may include an arithmetic logic unit. Exemplarily, processor 206 may include at least one adder, multiplier, etc. Exemplarily, processor 206 may include a multiply-accumulator. A multiply-accumulator can be used for multiply-accumulate operations to improve the processing efficiency of matrix multiplication, convolution, or circular convolution in deep learning models.

[0094] Figure 3 shows a schematic diagram of a processor consistent with some embodiments of this disclosure. Referring to Figure 3, in some embodiments, the processor 300 may include a plurality of processing units PE. 11 ~PE MN For example, the processing unit PE 11 ~PE MN The entire array is arranged in an array. Each processing unit PE... ij (1 < i ≤ M, 1 < j ≤ N) can be communicatively coupled to processing units in adjacent rows and columns. This enables rapid data transfer between processing units and improves data processing efficiency.

[0095] It should be noted that communication coupling can include one or more of the following methods: electrical connection, magnetic connection, optical connection, wireless connection, and wired connection. This disclosure does not limit the communication method between processing units.

[0096] For example, the wiring between the circuits in the processor 300 and the memory can be defined by burning a configuration file.

[0097] When the processor receives a point cloud calibration command from the controller, it can flexibly allocate processing units based on the echo data to be calibrated received from memory and the weights of the deep learning network. In this way, after the controller sends a point cloud calibration command, the processor can perform point cloud calibration on multiple channels. Compared to existing technologies that require multiple CPUs, the circuitry for point cloud calibration in some embodiments can save significant circuit area and improve circuit efficiency.

[0098] For different types of LiDAR, different echo datasets can be used to train the deep learning network. This yields weights for the deep learning network suitable for different LiDAR types. Based on these weights, the processor can flexibly call different processing units, or call them in different orders. This allows for point cloud calibration of different types of LiDAR without changing the hardware configuration.

[0099] In some embodiments, the processor 206 in the point cloud calibration circuit shown in FIG2 may include the processor 300 shown in FIG3.

[0100] In some embodiments, continuing to refer to FIG2, the circuit 200 for point cloud calibration may further include an analog-to-digital converter 208. The analog-to-digital converter 208 may be communicatively coupled to the memory 204. The analog-to-digital converter 208 may perform sampling and quantization operations on the echo signal received by the circuit according to a preset sampling frequency to obtain the echo data to be calibrated and store it in the memory 204.

[0101] The analog-to-digital converter 208 and the memory 204 can be coupled in one or more ways, such as electrical connection, optical connection, magnetic connection, wireless connection, or wired connection. For example, the analog-to-digital converter 208 and the memory 204 can transmit data via a bus to improve data transmission efficiency.

[0102] Figure 4 shows a schematic diagram of a lidar structure consistent with some embodiments of this disclosure. In some embodiments, referring to Figure 4, lidar 400 may include a laser 412, a detector 414, and a control and processing system 408.

[0103] Laser 412 can output detection signals to the outside.

[0104] Detector 414 can receive the echo signal returned after the detection signal is reflected by the object. Detector 414 can convert the reflected echo signal into an electrical signal using the photoelectric effect.

[0105] For example, laser 412 can emit laser light under the drive of a driving circuit (not shown in Figure 4). Control and processing system 408 can send control signals to the driving circuit to control the driving circuit to drive laser 412 to emit laser light. Under the control of control and processing system 408, laser 412 can emit laser light at a certain frequency, intensity, and emission time, forming a detection signal. When the detection signal encounters an object, it is reflected by the object. The light reflected back to lidar 400 can be called an echo. Detector 414 can convert the echo signal into an electrical signal. Based on the electrical signal converted by detector 414, echo data to be calibrated can be obtained. The echo data to be calibrated is provided to control and processing system 408.

[0106] In some embodiments, the control and processing system 408 may include a controller 402, a memory 404, and a processor 406.

[0107] The controller 402 can control the laser 412 to output detection signals and control the detector 414 to receive echo signals, as well as generate point cloud calibration instructions.

[0108] The memory 404 can store the echo data to be calibrated and the weights of the deep learning model.

[0109] The processor 406 can respond to point cloud calibration commands and retrieve the echo data to be calibrated and the weights of the deep learning model from the memory 404. The processor 406 can also invoke the deep learning model to calibrate the point cloud and obtain calibrated point cloud data.

[0110] The control and processing system 408 can first store the echo data to be calibrated in the memory 404. The memory 404 also stores the weights of the deep learning model. For example, the echo dataset can be input into a deep learning network for training to obtain the weights of the deep learning model and store them in the memory 404.

[0111] In some embodiments, the controller 402 can generate a point cloud calibration command when a preset trigger condition is met. For example, the controller 402 can generate point cloud calibration commands at a certain frequency. Another example is that the controller 402 can generate a point cloud calibration command when the amount of echo data to be calibrated stored in the memory 404 reaches a preset amount. Yet another example is that the controller 402 can generate a point cloud calibration command upon receiving a calibration indication signal. The point cloud calibration command can be output to the processor 406. In response to the point cloud calibration command, the processor 406 can retrieve the echo data to be calibrated and the weights of the deep learning model from the memory 404. The processor 406 can also invoke the deep learning model to calibrate the point cloud, obtaining calibrated point cloud data.

[0112] In some embodiments, there may be multiple lasers 412. There may also be multiple detectors 414. A lidar system may include multiple detection channels. Different detection channels can detect objects at different angles. A detection channel may include one or more lasers and one or more detectors. The correspondence between detection channels and lasers or detectors can be physically configured or configured by an algorithm. For example, one or more lasers included in a detection channel may have a preset correspondence with one or more detectors. When one or more lasers in a detection channel emit light, the corresponding one or more detectors receive the echo. As another example, a lidar system may include a detector array. One or more detectors at different positions on the detector array constitute pixels of a detection channel. Detectors within a pixel can receive echoes and output electrical signals. Pixels in different detection channels may include the same detectors. Pixels in different detection channels may also not include the same detectors.

[0113] In some embodiments, laser 412 may be the same as or similar to laser 112 in FIG. 1. Detector 414 may be the same as or similar to detector 114 in FIG. 1. Control and processing system 408 may be the same as or similar to control and processing system 108 in FIG. 1. LiDAR 400 may also include other structures, devices, or circuits related to the emission of laser light or the reception and processing of the echo, as can be seen in the example block diagram of the LiDAR shown in FIG. 1.

[0114] For example, processor 406 may include multiple processing units. These multiple processing units can execute deep learning models in parallel to calibrate the point cloud. This improves the efficiency of point cloud calibration.

[0115] In some embodiments, the echo dataset can be input into a deep learning network for training to obtain the weights of the deep learning model. Training the deep learning network with an echo dataset containing a large amount of echo data allows the weights of the deep learning model to more realistically and accurately reflect the biases in the echo data.

[0116] Echo datasets can include echo data from multiple channels. For example, echo data can include digital echo signals and echo information data. Information such as the detection angle or detection channel corresponding to the digital echo signal can be determined based on the echo information data. During training, the echo dataset can be input into a deep learning network to train weights for point cloud calibration.

[0117] In some embodiments, a deep learning model is trained after the LiDAR is assembled. The weights of the trained deep learning network are stored in a memory. Exemplarily, a deep learning model is trained separately for each LiDAR. The echo dataset used to train the deep learning model can be the same or different for different LiDARs.

[0118] In some embodiments, new echo data can be collected during the operation of the LiDAR. The echo dataset used for training the deep learning model can be updated. After the LiDAR has been running for a period of time, the deep learning model can be retrained using the updated echo dataset to obtain updated weights for the deep learning network. This allows for flexible responses to emerging data processing needs. The point cloud calibration function can be adjusted at the software level without requiring hardware changes.

[0119] In some embodiments, processor 406 may include an arithmetic logic unit. Exemplarily, processor 406 may include at least one adder, multiplier, etc. Exemplarily, processor 406 may include a multiply-accumulator. A multiply-accumulator can be used for multiply-accumulate operations to improve the processing efficiency of matrix multiplication, convolution, or circular convolution in deep learning models.

[0120] As in the previous example, processor 406 may include multiple processing units. In some embodiments, processor 406 may adopt the structure shown in processor 300 in FIG3. Continuing to refer to FIG3, processor 406 may include multiple processing units PE. 11 ~PE MN For example, multiple processing units PE 11 ~PE MN The entire array is arranged in an array. Each processing unit PE... ij (1 < i ≤ M, 1 < j ≤ N) can be communicatively coupled to processing units in adjacent rows and columns. This enables rapid data transfer between processing units and improves data processing efficiency.

[0121] In some embodiments, any processing unit PE ij This can include multiply-accumulators. Multiply-accumulators can be used for multiplication and addition operations to improve the processing efficiency of matrix multiplication, convolution, or circular convolution in deep learning models.

[0122] Referring again to Figure 4, when processor 406 receives a point cloud calibration command from controller 402, processor 406 can flexibly call upon processing units based on the echo data to be calibrated received from memory 404 and the weights of the deep learning network. Thus, after controller 402 sends a point cloud calibration command, processor 406 can perform point cloud calibration on multiple channels. Compared to existing technologies that require multiple CPUs, the circuitry for point cloud calibration in some embodiments can save significant circuit area and improve circuit efficiency.

[0123] For different types of LiDAR, different echo datasets can be used to train the deep learning network. This yields weights for the deep learning network suitable for different types of LiDAR. Based on these weights, the processor 406 can flexibly call different processing units, or call them in different orders. This allows for point cloud calibration of different types of LiDAR without changing the hardware configuration.

[0124] In some embodiments, continuing to refer to FIG4, the lidar 400 may further include an analog-to-digital converter 410. The analog-to-digital converter 410 may be communicatively coupled to a memory 404. The analog-to-digital converter 410 may perform sampling and quantization operations on the echo signal received by the detector 414 at a preset sampling frequency to obtain echo data to be calibrated. The analog-to-digital converter 410 may store the echo data to be calibrated in the memory 404.

[0125] In some embodiments, the analog-to-digital converter 410 and the memory 404 may be coupled in one or more ways, such as electrical connection, optical connection, magnetic connection, wireless connection, or wired connection. Exemplarily, the analog-to-digital converter 410 and the memory 404 may transmit data via a bus to improve data transmission efficiency. While the embodiments disclosed herein are as described above, the present invention is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the present invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A circuit for point cloud calibration, comprising: The controller is configured to generate point cloud calibration instructions; The memory is configured to store the echo data to be calibrated and the weights of the deep learning model; The processor is configured to, in response to the point cloud calibration command received from the controller, retrieve the echo data to be calibrated and the weights of the deep learning model from the memory, and calibrate the point cloud using the deep learning model to obtain calibrated point cloud data.

2. The circuit according to claim 1, characterized in that, The processor includes multiple processing units configured to execute the deep learning model to calibrate the point cloud.

3. The circuit according to claim 2, characterized in that, The plurality of processing units are arranged in an array, wherein any processing unit is communicatively coupled to the processing units in adjacent rows and columns.

4. The circuit according to any one of claims 1-3, characterized in that, The processing unit includes a multiply-accumulator.

5. The circuit according to claim 1, characterized in that, The echo dataset is input into the deep learning network for training, and the weights of the deep learning model are obtained.

6. The circuit according to claim 5, characterized in that, The deep learning network includes at least one of the following: a multilayer perceptron; a convolutional neural network; or a recurrent neural network.

7. The circuit according to claim 1, characterized in that, The point cloud calibration command is configured to characterize at least one of the following: filtering; or echo time or time-of-flight calibration.

8. The circuit according to claim 1, characterized in that, The memory includes static random access memory.

9. The circuit according to claim 1, characterized in that, Also includes: An analog-to-digital converter (ADC) is communicatively coupled to the memory. The ADC is configured to perform sampling and quantization operations on the echo signal received by the circuit at a preset sampling frequency to obtain the echo data to be calibrated and store it in the memory.

10. A lidar, comprising: A laser, configured to output a detection signal to the outside; The detector is configured to receive the echo signal returned after the detection signal is reflected by an object; The controller is configured to control the laser to output a detection signal and the detector to receive an echo signal, and to generate a point cloud calibration command; the memory is configured to store the echo data to be calibrated and the weights of the deep learning model. The processor is configured to respond to the point cloud calibration command, retrieve the echo data to be calibrated and the weights of the deep learning model from the memory, and call the deep learning model to calibrate the point cloud to obtain calibrated point cloud data.

11. The lidar according to claim 10, characterized in that, The processor includes multiple processing units configured to execute the deep learning model to calibrate the point cloud.

12. The lidar according to claim 11, characterized in that, The plurality of processing units are arranged in an array, wherein any processing unit is communicatively coupled to the processing units in adjacent rows and columns.

13. The lidar according to any one of claims 10-12, characterized in that, The processing unit includes a multiply-accumulator.

14. The lidar according to claim 10, characterized in that, The echo dataset is input into the deep learning network for training, and the weights of the deep learning model are obtained.

Images