Point cloud monitoring method, apparatus, and device

Abstract

The disclosure provides a point cloud monitoring method, device and equipment, relates to the technical field of image processing, specifically to the technical field of point cloud processing, and can be applied to an automatic driving scene. A specific embodiment of the method comprises the following steps: acquiring point cloud data collected by a laser radar to a field programmable logic gate array (FPGA); acquiring configuration parameters of a first monitoring item configured by an advanced reduced instruction set machine (ARM) to the FPGA; and processing the point cloud data using the configuration parameters on the FPGA to obtain a monitoring result of the first monitoring item. The embodiment is suitable for monitoring more point cloud data and can obtain a good monitoring result.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically point cloud processing technology, and can be applied to autonomous driving scenarios. Background Technology

[0002] A lidar system is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. Its working principle involves emitting a detection signal (laser beam) towards the target, then comparing the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, relevant information about the target can be obtained.

[0003] LiDAR enables higher levels of autonomous driving. The LiDAR monitoring system runs on top of the autonomous driving system, monitoring the point cloud data output by the LiDAR. When there are errors in the data frame rate or data content, the monitoring system reports an error. However, with the development of LiDAR circuitry, the amount of point cloud data that needs to be monitored is increasing. But the performance of FPGA (Field-Programmable Gate Array) and embedded ARM (Advanced RISC Machines) is limited and cannot handle such a large amount of point cloud data. Summary of the Invention

[0004] This disclosure provides a point cloud monitoring method, apparatus, device, storage medium, and program product.

[0005] In a first aspect, this disclosure proposes a point cloud monitoring method, comprising: acquiring point cloud data collected by a lidar onto a field-programmable gate array (FPGA); acquiring configuration parameters of a first monitoring item configured by a Reduced Instruction Set Machine (ARM) onto the FPGA; and processing the point cloud data on the FPGA using the configuration parameters to obtain the monitoring result of the first monitoring item.

[0006] Secondly, this disclosure provides a point cloud monitoring device, comprising: a first acquisition module configured to acquire point cloud data collected by a lidar onto a field-programmable gate array (FPGA); a second acquisition module configured to acquire configuration parameters of a first monitoring item configured by a Reduced Instruction Set Machine (ARM) onto the FPGA; and a processing module configured to process the point cloud data on the FPGA using the configuration parameters to obtain the monitoring result of the first monitoring item.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform a method as described in any implementation of the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0010] The point cloud monitoring method provided in this disclosure is suitable for monitoring larger amounts of point cloud data and can achieve good monitoring results. It will not cause point cloud data loss, nor will it trigger frame rate error alarms.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart of one embodiment of the point cloud monitoring method according to the present disclosure;

[0014] Figure 2 This is a flowchart of yet another embodiment of the point cloud monitoring method according to the present disclosure;

[0015] Figure 3 This is a scene diagram that can implement the point cloud monitoring method of the embodiments of this disclosure;

[0016] Figure 4 This is a schematic diagram of a structure of an embodiment of the point cloud monitoring device according to the present disclosure;

[0017] Figure 5 This is a block diagram of an electronic device used to implement the point cloud monitoring method of the embodiments of this disclosure. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1 A flow 100 of an embodiment of a point cloud monitoring method according to the present disclosure is shown. The point cloud monitoring method includes the following steps:

[0021] Step 101: The point cloud data collected by the lidar is acquired and transferred to the field programmable gate array (FPGA).

[0022] In this embodiment, the lidar can collect point cloud data and send it to the FPGA.

[0023] Typically, autonomous vehicles are equipped with LiDAR (Light Detection and Ranging) sensors. When the vehicle is in autonomous driving mode, the LiDAR can collect data about its surroundings, generating a large amount of point cloud data. Furthermore, with the development of LiDAR technology, the amount of point cloud data collected is increasing. This point cloud data can first be sent to an FPGA (FPGA-based system for preprocessing).

[0024] Step 102: Obtain the configuration parameters of the first monitoring item of the ARM Advanced Reduced Instruction Set Machine configuration onto the FPGA.

[0025] In this embodiment, the FPGA can obtain the configuration parameters of the first monitoring item configured by the ARM CPU. The first monitoring item can be a monitoring item that needs to be statistically analyzed on the FPGA. The first monitoring item can monitor for anomalies in at least one of the following: distance, angle, reflectivity, and frame rate of the point cloud data, such as CRC (Cyclic Redundancy Check) or error frame statistics.

[0026] Typically, configuration parameters written directly to the FPGA cannot be modified. To make modifications to the first monitoring item more flexible, configuration parameters for the first monitoring item can be configured on the ARM CPU. When statistics for the first monitoring item need to be collected on the FPGA, the corresponding configuration parameters can be retrieved from the ARM CPU. Since the ARM CPU supports flexible modification of configuration parameters, when a new monitoring item needs to be added, the number of items that the FPGA can monitor can be increased simply by adding the corresponding configuration parameters on the ARM CPU.

[0027] Step 103: Process the point cloud data on the FPGA using configuration parameters to obtain the monitoring results of the first monitoring project.

[0028] In this embodiment, the FPGA can use the configuration parameters of the first monitoring item obtained from the ARM CPU to process the point cloud data and obtain the monitoring results of the first monitoring item.

[0029] The first monitoring item can be a monitoring item that needs to be counted on the FPGA, such as CRC or error frame statistics, and its monitoring results can be such as CRC check results or error frames.

[0030] Typically, the FPGA obtains the corresponding configuration parameters from the ARM CPU to process the point cloud data, based on the configuration parameters of the first monitored item configured on the ARM CPU.

[0031] The point cloud monitoring method provided in this disclosure is suitable for monitoring larger amounts of point cloud data and can achieve good monitoring results. It will not cause point cloud data loss, nor will it trigger frame rate error alarms.

[0032] Continue to refer to Figure 2 This illustrates a flow 200 of yet another embodiment of the point cloud monitoring method according to the present disclosure. The point cloud monitoring method includes the following steps:

[0033] Step 201: The point cloud data collected by the lidar is acquired and transferred to the field programmable gate array (FPGA).

[0034] In this embodiment, the lidar can collect point cloud data and send it to the FPGA.

[0035] Typically, autonomous vehicles are equipped with LiDAR (Light Detection and Ranging) sensors. When the vehicle is in autonomous driving mode, the LiDAR can collect data about its surroundings, generating a large amount of point cloud data. Furthermore, with the development of LiDAR technology, the amount of point cloud data collected is increasing. This point cloud data can first be sent to an FPGA (FPGA-based system for preprocessing).

[0036] Step 202: Obtain the configuration parameters of the first monitoring item of the ARM Advanced Reduced Instruction Set Machine configuration onto the FPGA.

[0037] In this embodiment, the FPGA can obtain the configuration parameters of the first monitoring item configured by the ARM CPU. The first monitoring item can be a monitoring item that needs to be statistically analyzed on the FPGA. The first monitoring item can monitor for anomalies in at least one of the following: distance, angle, reflectivity, and frame rate of the point cloud data, such as CRC or error frame statistics.

[0038] Typically, configuration parameters written directly to the FPGA cannot be modified. To make modifications to the first monitoring item more flexible, configuration parameters for the first monitoring item can be configured on the ARM CPU. When statistics for the first monitoring item need to be collected on the FPGA, the corresponding configuration parameters can be retrieved from the ARM CPU. Since the ARM CPU supports flexible modification of configuration parameters, when a new monitoring item needs to be added, the number of items that the FPGA can monitor can be increased simply by adding the corresponding configuration parameters on the ARM CPU.

[0039] Step 203: Process the point cloud data on the FPGA using configuration parameters to obtain the monitoring results of the first monitoring project.

[0040] In this embodiment, the FPGA can use the configuration parameters of the first monitoring item obtained from the ARM CPU to process the point cloud data and obtain the monitoring results of the first monitoring item.

[0041] The first monitoring item can be a monitoring item that needs to be counted on the FPGA, such as CRC or error frame statistics, and its monitoring results can be such as CRC check results or error frames.

[0042] Typically, the FPGA obtains the corresponding configuration parameters from the ARM CPU to process the point cloud data, based on the configuration parameters of the first monitored item configured on the ARM CPU.

[0043] Step 204: Send the monitoring results of the first monitoring item on the FPGA to the ARM.

[0044] In this embodiment, the FPGA can send the monitoring results of the first monitoring item to the OCM (On-Chip Memory) of the ARM CPU. Since the data volume of the monitoring results of the first monitoring item is much smaller than that of the point cloud data, even if the performance of the FPGA embedded ARM is limited, it can still complete the continued statistical analysis of the monitoring results of the first monitoring item with a small data volume.

[0045] To quickly transfer data, the FPGA can send the monitoring results of the first monitoring item to the ARM CPU's OCM via AXI-DMA (Direct Memory Access). DMA is a memory access technology that allows certain internal computer hardware subsystems to independently and directly read and write memory without CPU intervention, thus reducing the CPU's interrupt load. Otherwise, the CPU would need to copy each segment of data from its source to registers and then write it back to its new location, during which time the CPU cannot perform other tasks. Furthermore, DMA is a fast data transfer method, typically used to transfer large blocks of data. When using DMA, the CPU sends a memory transfer request to the DMA controller. While the DMA controller is transferring data, the CPU performs other operations, and the DMA notifies the CPU of the transfer completion via an interrupt.

[0046] Step 205: Perform statistics on the monitoring results of the first monitoring item on the ARM to obtain the monitoring results of the second monitoring item.

[0047] In this embodiment, the ARM CPU can perform statistics based on the monitoring results of the first monitoring item to obtain the monitoring results of the second monitoring item.

[0048] In this embodiment, the FPGA can send the monitoring results of the first monitoring item to the OCM (On-Chip Memory) of the ARM CPU. Since the data volume of the monitoring results of the first monitoring item is much smaller than that of the point cloud data, even if the performance of the FPGA embedded ARM is limited, it can still complete the continued statistical analysis of the monitoring results of the first monitoring item with a small data volume.

[0049] from Figure 2 It can be seen from this that, with Figure 1 Compared to the corresponding embodiments, the point cloud monitoring method in this embodiment adds an ARM monitoring step to process flow 200. Therefore, the monitoring results of the first monitoring item based on the FPGA in the scheme described in this embodiment are used to continue monitoring other items. Since the data volume of the monitoring results of the first monitoring item is much smaller than the point cloud data, even with limited performance of the FPGA embedded ARM, it is still possible to complete the continued statistical analysis of the monitoring results of the first monitoring item, which has a smaller data volume.

[0050] For ease of understanding, Figure 3 A scene diagram is shown that can implement the point cloud monitoring method of the embodiments of this disclosure. For example... Figure 3As shown, three LiDAR sensors collect point cloud data and send it to the FPGA via a switch. The FPGA uses the ARM CPU's configuration parameter file to perform CRC checks or error frame statistics on the point cloud data, obtaining the monitoring results. The FPGA then sends the monitoring results to the ARM CPU's OCM via AXI-DMA. The ARM CPU continues to perform statistics on other monitoring items based on the monitoring results.

[0051] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a point cloud monitoring device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0052] like Figure 4 As shown, the point cloud monitoring device 400 of this embodiment may include: a first acquisition module 401, a second acquisition module 402, and a processing module 403. The first acquisition module 401 is configured to acquire point cloud data collected by a LiDAR and load it onto a Field Programmable Gate Array (FPGA); the second acquisition module 402 is configured to acquire configuration parameters of a first monitoring item configured by a Reduced Instruction Set Machine (ARM) and load them onto the FPGA; the processing module 403 is configured to process the point cloud data on the FPGA using the configuration parameters to obtain the monitoring results of the first monitoring item.

[0053] In this embodiment, the specific processing of the first acquisition module 401, the second acquisition module 402, and the processing module 403 in the point cloud monitoring device 400, and the resulting technical effects, can be referred to respectively. Figure 2 The relevant descriptions of steps 101-103 in the corresponding embodiments will not be repeated here.

[0054] In some optional implementations of this embodiment, the point cloud monitoring device 400 further includes: a sending module configured to send the monitoring results of the first monitoring item on the FPGA to the ARM; and a statistics module configured to perform statistics on the ARM based on the monitoring results of the first monitoring item to obtain the monitoring results of the second monitoring item.

[0055] In some optional implementations of this embodiment, the sending module is further configured to send the monitoring results of the first monitoring item on the FPGA to the ARM via direct memory access AXI-DMA.

[0056] In some optional implementations of this embodiment, the items monitored by the FPGA are increased by adding configuration parameters on the ARM.

[0057] In some optional implementations of this embodiment, the first monitoring item includes Cyclic Redundancy Check (CRC) or error frame statistics.

[0058] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0059] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0060] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0061] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0062] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0063] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the point cloud monitoring method. For example, in some embodiments, the point cloud monitoring method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the point cloud monitoring method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the point cloud monitoring method by any other suitable means (e.g., by means of firmware).

[0064] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0065] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0066] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0067] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0068] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0069] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0070] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0071] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A point cloud monitoring method, comprising: The point cloud data collected by the lidar is acquired and processed onto a field-programmable gate array (FPGA). The configuration parameters of the first monitoring item of the ARM configuration are obtained and transferred to the FPGA. The FPGA monitoring items are added by adding configuration parameters on the ARM. The first monitoring item includes cyclic redundancy check (CRC) or error frame statistics. The point cloud data is processed using the configuration parameters on the FPGA to obtain the monitoring results of the first monitoring item; The monitoring results of the first monitoring item on the FPGA are sent to the ARM. The monitoring results of the second monitoring item are obtained by performing statistics on the ARM based on the monitoring results of the first monitoring item.

2. The method according to claim 1, wherein, Sending the monitoring results of the first monitoring item on the FPGA to the ARM includes: The monitoring results of the first monitoring item on the FPGA are sent to the ARM via Direct Memory Access AXI-DMA.

3. A point cloud monitoring device, comprising: The first acquisition module is configured to acquire point cloud data collected by the lidar onto a field-programmable gate array (FPGA). The second acquisition module is configured to acquire the configuration parameters of the first monitoring item of the ARM configuration of the ARM and transfer them to the FPGA. The FPGA monitors the items by adding configuration parameters on the ARM. The first monitoring item includes cyclic redundancy check (CRC) or error frame statistics. The processing module is configured to process the point cloud data on the FPGA using the configuration parameters to obtain the monitoring results of the first monitoring item. The sending module is configured to send the monitoring results of the first monitoring item on the FPGA to the ARM. The statistics module is configured to perform statistics on the ARM based on the monitoring results of the first monitoring item to obtain the monitoring results of the second monitoring item.

4. The apparatus according to claim 3, wherein, The sending module is further configured to: The monitoring results of the first monitoring item on the FPGA are sent to the ARM via Direct Memory Access AXI-DMA.

5. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-2.

6. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-2.

7. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-2.

Images