A method, apparatus, electronic device and system for real-time rendering of point clouds

Abstract

The application relates to the technical field of image processing, in particular to a method, device, electronic equipment and system for real-time rendering of point clouds. The method for real-time rendering of point clouds comprises the following steps: acquiring point cloud data by using data collected during scanning by a laser radar, and initializing the point cloud data to obtain initialized point cloud data; loading the initialized point cloud data in parallel, calling a Unity engine to convert a point cloud array into mesh vertices in parallel; and performing affine transformation on each vertex coordinate in the mesh vertices to realize rendering of the point clouds. The application can greatly reduce the performance overhead of CPU and memory, thereby improving the real-time rendering frame rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of laser radar-based environment perception technology, and in particular to a method, device, electronic equipment and system for real-time rendering of point clouds. BACKGROUND

[0002] Laser radar is the abbreviation of laser detection and ranging system. Laser radar determines the propagation distance between the sensor or emitter and the target object, analyzes the information such as the reflected energy size, the amplitude, frequency and phase of the reflected wave spectrum of the target object surface, and thus presents the accurate three-dimensional (3D) structure information of the target object. Due to its high positioning accuracy, easy-to-implement algorithm, and less influence by day and night, etc., laser radar has become the core device for automatic driving positioning and detection.

[0003] A point cloud is a collection of mass points expressing the distribution of a target space and the characteristics of a target surface in the same space reference system. After obtaining the spatial coordinates of each sampling point of an object surface, a collection of points is obtained. Point clouds are often used to describe 3D scenes, and there are various ways to generate them. Among them, laser radar point clouds are generated after post-processing of laser radar scanning. In the fields of automatic driving and remote sensing, laser radar point clouds play an important role, for example, they can be used to generate high-precision maps for automatic driving. How to perform real-time rendering of point clouds with high performance has become an important part of the visualization system for automatic driving. SUMMARY

[0004] In view of this, the present application provides a method, device, electronic equipment and system for high-performance real-time rendering of point clouds.

[0005] In a first aspect, an embodiment of the present application provides a method for real-time rendering of point clouds, comprising:

[0006] acquiring point cloud data using data collected during laser radar scanning, and initializing the point cloud data to obtain initialized point cloud data;

[0007] loading the initialized point cloud data in parallel, and converting the point cloud array into mesh vertices in parallel by calling a Unity engine;

[0008] performing affine transformation on the coordinates of each vertex in the mesh vertices to realize rendering of the point clouds.

[0009] The method for real-time rendering of point clouds provided by the embodiment uses the rendering method based on mesh vertices in the Unity engine, which can greatly reduce the performance overhead of CPU and memory, thereby improving the real-time rendering frame rate and providing a better solution for rendering of point clouds.

[0010] As an implementation form of the first aspect, the parallel loading of the initialized point cloud data, the calling of the Unity engine to convert the point cloud array into mesh vertices in parallel, comprises:

[0011] transmitting the initialized point cloud data into a storage area of the Unity engine after being converted into the point cloud array;

[0012] parallel processing the point cloud array in the storage area to obtain mesh vertices of the point cloud array, and transmitting the mesh vertices to the GPU.

[0013] As an implementation form of the first aspect, the parallel loading of the initialized point cloud data, the calling of the Unity engine to convert the point cloud array into mesh vertices in parallel, comprises:

[0014] transmitting the initialized point cloud data into a shared memory of the Unity engine after being converted into the point cloud array;

[0015] parallel processing the point cloud array in the shared memory to obtain mesh vertices of the point cloud array, and transmitting the mesh vertices to the GPU.

[0016] The method for rendering point cloud in real time provided by the implementation form stores the point cloud array in the shared memory, so that the main thread and each job thread can access the shared memory, the overhead of memory copying is avoided, and the scheme is further optimized.

[0017] As an implementation form of the first aspect, the parallel loading of the initialized point cloud data, the calling of the Unity engine to convert the point cloud array into mesh vertices in parallel, comprises:

[0018] transmitting the initialized point cloud data into a calculation buffer area of the Unity engine after being converted into the point cloud array;

[0019] processing the point cloud array in the calculation buffer area to obtain mesh vertices of the point cloud array, and transmitting the mesh vertices to the GPU.

[0020] As an implementation form of the first aspect, the affine transformation of each vertex coordinate in the mesh vertices to realize the rendering of the point cloud comprises:

[0021] stretching each vertex coordinate in the mesh vertices by using a shader of the Unity engine in the GPU, calculating new each vertex coordinate after the stretching, and rendering the point cloud onto a pixel screen according to the new each vertex coordinate.

[0022] As an implementation form of the first aspect, the initialization of the point cloud data to obtain the initialized point cloud data comprises:

[0023] initializing the point cloud data according to a preset format to obtain the point cloud data in the preset format.

[0024] As an implementation form of the first aspect, the point cloud data is acquired by using data collected during laser radar scanning, including:

[0025] The depth image and basic data collected during laser radar scanning are acquired, and the point cloud data is acquired according to the depth image and the basic data.

[0026] As an implementation form of the first aspect, the basic data includes a scanning angle and a time stamp when the depth image is collected.

[0027] In a second aspect, an embodiment of the present application provides a device for rendering a point cloud in real time, including:

[0028] An initialization module is configured to acquire point cloud data by using data collected during laser radar scanning, and to initialize the point cloud data to obtain initialized point cloud data.

[0029] A conversion module is configured to load the initialized point cloud data in parallel, and to convert the point cloud array into mesh vertices in parallel.

[0030] A rendering module is configured to perform affine transformation on each vertex coordinate in the mesh vertices to realize point cloud rendering.

[0031] The conversion module and the rendering module can call a Unity engine to work.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for rendering a point cloud in real time according to the first aspect or any implementation form of the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a system for rendering a point cloud in real time, including one or more laser radars and an electronic device according to the third aspect, and the one or more laser radars are connected to the electronic device.

[0034] As an implementation form of the fourth aspect, the system for rendering a point cloud in real time includes an automatic driving system or a remote sensing telemetry system.

[0035] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method for rendering a point cloud in real time according to the first aspect or any implementation form of the first aspect.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when running on an electronic device, causes the electronic device to perform the method for real-time rendering of a point cloud according to the first aspect or any implementation manner of the first aspect.

[0037] It can be understood that the beneficial effects of the second aspect to the sixth aspect can be referred to the related description in the first aspect and any implementation manner of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0039] Figure 1 is a structural schematic diagram of a system for real-time rendering of a point cloud provided by an embodiment of the present application;

[0040] Figure 2 is an implementation flowchart of a method for real-time rendering of a point cloud provided by an embodiment of the present application;

[0041] Figure 3 is a process schematic diagram of a method for real-time rendering of a point cloud provided by an embodiment of the present application;

[0042] Figure 4 is a specific implementation flowchart of step S110 in the method for real-time rendering of a point cloud provided by an embodiment of the present application;

[0043] Figure 5 is a specific implementation flowchart of step S120 in the method for real-time rendering of a point cloud provided by an embodiment of the present application

[0044] Figure 6 is an implementation flowchart of another method for real-time rendering of a point cloud provided by an embodiment of the present application;

[0045] Figure 7 is a structural schematic diagram of a device for real-time rendering of a point cloud provided by an embodiment of the present application;

[0046] Figure 8 is a structural schematic diagram of an initialization module in the device for real-time rendering of a point cloud provided by an embodiment of the present application;

[0047] Figure 9 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0049] The term "and / or" as used in the specification and in the claims, means any one of the associated listed items, or a combination of any of the associated listed items, and includes all possible combinations.

[0050] "one or more embodiments" or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in yet other embodiments", or the like, in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specifically specified. The terms "comprises", "comprising", "has", "having", "includes", "including", "contains", "containing" or variations thereof, are intended to mean "including but not limited to", unless otherwise specifically specified.

[0051] Furthermore, in the description of the application, the meaning of "a plurality" is two or more. The terms "first", "second", "third", and "fourth" and the like are used to distinguish descriptions and are not to be understood as indicating or implying relative importance.

[0052] In order to illustrate the technical solutions described in the present application, the following will be described by specific embodiments.

[0053] In the field of large-scale data visualization, the speed of graphic rendering is one of the most important factors for application development. Since large-scale data visualization usually requires three-dimensional representation, the Unity engine, as the current mainstream 3D visualization engine, is widely used in various 3D rendering implementations.

[0054] For example, an autonomous driving system usually uses a 64-line laser radar, which generates a large number of point clouds per second, reaching 1.2 million. If the transmission frequency is 10Hz, in the Unity engine, about 120,000 point cloud data need to be rendered per frame. The traditional rendering method is usually to use the particle system of the Unity engine, to correspond each point in the point cloud to each particle, and to assign the corresponding coordinates and color to the particle. However, the large-scale use of particle effects will cause high Draw call, large rendering overhead, and high CPU occupation.

[0055] The embodiment of the present application provides a method, device and electronic equipment for real-time rendering of point cloud, uses a mesh filter (MeshFilter) of a Unity engine to render the point cloud, can greatly reduce the performance overhead of CPU and memory, thereby improving the real-time rendering frame rate, and provides a more optimal point cloud rendering solution.

[0056] The method for real-time rendering of point cloud provided by the embodiment of the present application can be applied to electronic equipment. The electronic equipment includes but is not limited to a computer, a tablet computer, a vehicle-mounted terminal or a server, and the like. The server includes but is not limited to a standalone server or a cloud server, and the like.

[0057] In some embodiments, the electronic equipment and the laser radar are components of an automatic driving system or a remote sensing telemetry system. As shown in Figure 1 The system for real-time rendering of point cloud, for example, a structural schematic diagram of an automatic driving system or a remote sensing telemetry system, includes an electronic equipment 11 and a laser radar 12. The electronic equipment 11 is connected to the laser radar 12 in a wired or wireless manner.

[0058] It should be noted that the laser radar can be one or more according to different actual use scenarios, and a single-line laser radar and / or a multi-line laser radar can be used. The system can further include more components, Figure 1 which are only described as examples.

[0059] Figure 2 is an implementation flow diagram of the method for real-time rendering of point cloud provided by an embodiment of the present application, Figure 3 is an implementation process diagram of the method for real-time rendering of point cloud provided by an embodiment of the present application. The method for real-time rendering of point cloud provided by the embodiment of the present application is suitable for a case of visualizing after rendering point cloud data. The method is executed by a device for real-time rendering of point cloud, and the device for real-time rendering of point cloud is usually integrated in an electronic equipment, for example Figure 1 The electronic equipment 11 shown in

[0060] As shown in Figure 2 The method for real-time rendering of point cloud can include steps S110 to S130.

[0061] S110, obtaining point cloud data by using data collected when the laser radar is scanned, and initializing the point cloud data to obtain initialized point cloud data.

[0062] In some embodiments, as shown in Figure 1As shown in FIG. 1, the electronic device 11 is connected to the laser radar 12. The data collected by the laser radar during scanning includes not only the depth image, but also basic data such as the scanning angle and timestamp when the current depth image is collected. The electronic device acquires the depth image and basic data collected by the laser radar at a preset frequency. The preset frequency can be a system default setting or a user-defined setting. The electronic device obtains point cloud data by using the depth image and basic data, and initializes the point cloud data to obtain initialized point cloud data.

[0063] As a non-limiting example, as shown in FIG. 2, the step S110 specifically includes steps S111 and S112. Figure 4 As shown in FIG. 1, the electronic device 11 is connected to the laser radar 12. The data collected by the laser radar during scanning includes not only the depth image, but also basic data such as the scanning angle and timestamp when the current depth image is collected. The electronic device acquires the depth image and basic data collected by the laser radar at a preset frequency. The preset frequency can be a system default setting or a user-defined setting. The electronic device obtains point cloud data by using the depth image and basic data, and initializes the point cloud data to obtain initialized point cloud data.

[0064] S111, obtaining point cloud data by using the data collected by the laser radar during scanning.

[0065] In some embodiments, the data collected by the laser radar during scanning includes the depth image, and also includes basic data such as the scanning angle and timestamp when the current depth image is collected.

[0066] As a non-limiting example, as shown in FIG. 2, the step S110 specifically includes steps S111 and S112. Figure 3 As shown in FIG. 1, the electronic device 11 is connected to the laser radar 12. The data collected by the laser radar during scanning includes not only the depth image, but also basic data such as the scanning angle and timestamp when the current depth image is collected. The electronic device acquires the depth image and basic data collected by the laser radar at a preset frequency. The preset frequency can be a system default setting or a user-defined setting. The electronic device obtains point cloud data by using the depth image and basic data, and initializes the point cloud data to obtain initialized point cloud data.

[0067] More specifically, the calculation method for converting the depth image collected by the laser radar into point cloud data is as follows:

[0068]

[0069] wherein (x s ,y s ,z s ) is the three-dimensional coordinates of the point cloud in the laser radar camera coordinate system, z is the depth on each pixel, (u,v) is the pixel coordinate, (u0,v0) is the principal point coordinate of the image, dx and dy are the physical sizes of the sensor pixels of the laser radar in two directions, and f’ is the focal length (unit: millimeter).

[0070] It should be noted that the point cloud data corresponding to each pixel is also assigned with basic data such as the scanning angle and timestamp when each frame of depth image is collected by the laser radar during scanning.

[0071] S112, initializing the point cloud data to obtain point cloud data in a preset format.

[0072] The point cloud data obtained in step S110 is initialized according to a preset format to obtain point cloud data in the preset format. For example, the point cloud data can be converted into a polygon file format (PLY), a PCD file format, or a txt file format. It should be noted that this application embodiment does not impose specific limitations on the format of the point cloud data, and point cloud data in any format is applicable to this application.

[0073] In some embodiments, point cloud data is converted into a PLY file format, which can be used to describe a polygonal model. The PLY file format includes a header and an element list. The header includes a description of each element type, the element name, the number of elements, and a list of different attributes associated with that element. Element names include, for example, edges, vertices, and faces. The element list includes a vertex list, a face list, and other element lists.

[0074] As a non-limiting example, a polygon model can be described by elements such as the vertices of the point cloud, the faces formed between the vertices, and a list of elements, along with associated attributes such as scan angles and timestamps. The list of elements includes the three-dimensional coordinates of all vertices in the point cloud data.

[0075] S120 loads the initialized point cloud data in parallel and calls the Unity engine to convert the initialized point cloud data into mesh vertices in parallel.

[0076] In some embodiments, initialized point cloud data is obtained, the initialized point cloud array is loaded in parallel, and the Unity engine is invoked to convert the initialized point cloud data into mesh vertices in parallel. Mesh vertices include data of each vertex of the mesh in the Unity engine, such as coordinate data.

[0077] As a non-restrictive example, the Unity engine is invoked to convert point cloud data in PLY file format into data for each vertex of the Mesh in the Unity engine in parallel.

[0078] In some embodiments, such as Figure 5 As shown, step S120 specifically includes steps S121 to S122.

[0079] S121 converts the initialized point cloud data into a point cloud array and then transmits it to the shared memory of the Unity engine.

[0080] In some implementations, the initialized point cloud data is loaded, the Unity engine is called to convert the initialized point cloud data into a point cloud array, and then the point cloud array is transferred to the Unity engine's shared memory. See [link to relevant documentation]. Figure 3The main thread and each job can access the shared memory, avoiding the overhead of memory copying.

[0081] As a non-limiting example, the shared memory can include a shared memory type Native Container. The Native Container is a container provided by Unity for memory sharing between C# and Native, avoiding the overhead of memory copying.

[0082] In some other embodiments, further, after step S121, each point cloud array can be assigned a corresponding index according to its address in the shared memory, so that it can be better parallel processed in subsequent processing. For example, as shown in FIG. 1 1, the address of the lidar point cloud array a in the shared memory is NativeArray_a; the address of the lidar point cloud array b in the shared memory is NativeArray_b. Figure 3

[0083] S122, parallel processing the point cloud arrays in the shared memory to obtain the mesh vertices of the point cloud arrays, and transmitting the mesh vertices to the GPU.

[0084] In some embodiments, the point cloud arrays in the shared memory can be parallel processed by the job system in the Unity engine to obtain the mesh vertices of the point cloud arrays, and the mesh vertices are transmitted to the GPU. By using the shared memory type Native Container and the job system to realize parallel loading of point cloud data, the overall performance of the scheme can be improved.

[0085] Since obtaining the mesh vertices and transmitting the vertex data to the GPU are asynchronous logical instructions, and the Unity engine usually has one worker thread per CPU core, the jobs are synchronously scheduled among these threads. Therefore, the asynchronous logical instructions can be encapsulated as separate elements by the job system in the Unity engine, and appropriate threads are scheduled by the Unity engine, avoiding direct thread operation, and thus avoiding spending a lot of time cost.

[0086] As a non-limiting example, continuing to refer to FIG. 1 1, Figure 3 ​As shown, the mesh rendering interface Mesh.SetVertices of the Unity engine is called to convert the point cloud array into mesh vertices, and Mesh.UploadMeshData is called to immediately transmit the set mesh vertex data to the GPU. Since the above process is an asynchronous instruction, the above process can be encapsulated as multiple individual elements by the job system, and multiple point cloud arrays can be read simultaneously from the Native Container container by multi-threading for parallel processing to obtain multiple mesh vertices and load the multiple mesh vertices from the CPU to the GPU in parallel.

[0087] It should be noted that the correspondence between the Unity engine and the number of CPU cores can be controlled to achieve optimal running effect.

[0088] S130, affine transformation is performed on the coordinates of each vertex in the mesh vertex to realize point cloud rendering.

[0089] In some embodiments, continuing to refer to Figure 3 As shown, the vertex coordinates are stretched in the GPU using the vertex shader of the Unity engine, and the new vertex coordinates after stretching are calculated, and finally the point cloud is rendered onto the pixel screen according to the new vertex coordinates.

[0090] Specifically, when the mesh vertex is imported into the Unity engine, the vertex coordinates of the model can be accessed in the vertex shader, which contains the coordinates of each vertex.

[0091] The first step of vertex transformation is to transform the vertex coordinates from the model space to the world space, which is usually called model transformation.

[0092] The second step of vertex transformation is to transform the vertex coordinates from the world space to the observation space, which is called view transformation.

[0093] The third step is to convert the vertex from the observation space to the clipping space, and the matrix used for transformation is called clipping matrix / projection matrix. Projection is actually a dimension reduction from four-dimensional space to two-dimensional screen space.

[0094] The fourth step is to perform clipping work after the transformation of the projection matrix. After all the clipping work is completed, the real projection is performed to obtain the real pixel position, rather than the virtual three-dimensional coordinates.

[0095] It should be noted that in other embodiments, a surface shader (SurfaceShaders) or a fixed function shader (Fixed Function Shaders) of the Unity engine can also be called to complete rendering of the mesh vertex.

[0096] The embodiment of the application realizes high-performance real-time rendering of point clouds by fully utilizing the mesh filter component in the Unity engine and combining CPU multi-threading technology. Compared with the traditional particle system rendering method, the scheme has the advantages of low Drawcall, small rendering overhead, and high frame rate.

[0097] Figure 6 A method for real-time rendering of point clouds is provided for another embodiment of the application. It should be understood that the embodiment is the same as the foregoing embodiment, and thus will not be described again.

[0098] As shown in Figure 6 A method for real-time rendering of point clouds includes steps S210 to S240.

[0099] S210, obtaining point cloud data by using data collected during laser radar scanning, and initializing the point cloud data to obtain initialized point cloud data.

[0100] S220, converting the initialized point cloud data into a point cloud array and transmitting the point cloud array to a compute buffer of the Unity engine.

[0101] In some embodiments, the Unity engine is called to convert the initialized point cloud data into a point cloud array in parallel, and the point cloud array is transmitted to the compute buffer of the Unity engine.

[0102] S230, processing the point cloud array in the compute buffer in parallel to obtain mesh vertices of the point cloud array, and transmitting the mesh vertices to the GPU.

[0103] S240, performing affine transformation on the coordinates of each vertex in the mesh vertices to realize point cloud rendering.

[0104] The embodiment of the application differs from the foregoing embodiment in that, instead of sharing memory, the compute buffer (computeBuffer) is used to store the point cloud array, and each point cloud array cannot be processed in parallel, which results in a larger memory overhead compared with the foregoing embodiment.

[0105] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0106] An embodiment of the present application further provides a device for rendering a point cloud in real time. Details not described in the device for rendering a point cloud in real time are described in the foregoing method embodiments.

[0107] Referring to Figure 7 , Figure 7 is a schematic block diagram of a device for rendering a point cloud in real time provided by an embodiment of the present application. The device for rendering a point cloud in real time comprises an initialization module 71, a conversion module 72 and a rendering module 73.

[0108] The initialization module 71 is configured to obtain point cloud data by using data collected during laser radar scanning, and to initialize the point cloud data to obtain initialized point cloud data.

[0109] The conversion module 72 is configured to load the initialized point cloud data in parallel, and to convert a point cloud array into mesh vertices in parallel.

[0110] The rendering module 73 is configured to perform affine transformation on each vertex coordinate in the mesh vertices to realize point cloud rendering.

[0111] In some embodiments, the conversion module 72 and the rendering module 73 can call a Unity engine to work, specifically:

[0112] The initialized point cloud data is converted into a point cloud array and then transmitted to a storage area of the Unity engine;

[0113] The point cloud array in the storage area is processed in parallel to obtain mesh vertices of the point cloud array, and the mesh vertices are transmitted to a GPU.

[0114] As an implementation manner, the conversion module 72 is specifically configured to:

[0115] The initialized point cloud data is converted into a point cloud array and then transmitted to a shared memory of the Unity engine;

[0116] The point cloud array in the shared memory is processed in parallel to obtain mesh vertices of the point cloud array, and the mesh vertices are transmitted to a GPU.

[0117] As another implementation manner, the conversion module 72 is specifically configured to:

[0118] The initialized point cloud data is converted into a point cloud array and then transmitted to a calculation buffer area of the Unity engine;

[0119] The point cloud array in the calculation buffer area is processed in parallel to obtain mesh vertices of the point cloud array, and the mesh vertices are transmitted to a GPU.

[0120] In some embodiments, the rendering module 73 is specifically configured to:

[0121] The Unity engine is used in the GPU to stretch each vertex coordinate in the mesh vertex, and to calculate new each vertex coordinate after stretching, and to render the point cloud to the pixel screen according to the new each vertex coordinate.

[0122] In some embodiments, as shown in Figure 8 The initialization module 71 includes an acquisition sub-module 711 and an initialization sub-module 712.

[0123] The acquisition sub-module 711 is configured to acquire a depth image and basic data thereof collected during laser radar scanning, and to acquire point cloud data according to the depth image and the basic data thereof.

[0124] The initialization sub-module 712 is configured to initialize the point cloud data in a preset format, and to obtain point cloud data in the preset format.

[0125] An embodiment of the present application further provides an electronic device, as shown in Figure 9 The electronic device can include one or more processors 90 (only one processor is shown in Figure 9 The memory 91 and the computer program 92 stored in the memory 91 and executable on the one or more processors 90, for example, a program for rendering a point cloud in real time. The one or more processors 90 can implement each step in the method embodiment of rendering a point cloud in real time when executing the computer program 92. Alternatively, the one or more processors 90 can implement the functions of each module / unit in the device embodiment of rendering a point cloud in real time when executing the computer program 92, which is not limited here.

[0126] Those skilled in the art can understand, Figure 9 is merely an example of the electronic device and does not constitute a limitation on the electronic device. The electronic device can include more or fewer components than those shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, etc.

[0127] In an embodiment, the processor 90 can be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can be any conventional processor.

[0128] In an embodiment, the memory 91 can be an internal memory unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory 91 can also be an external memory device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like equipped in the electronic device. Further, the memory 91 can include both the internal memory unit and the external memory device of the electronic device. The memory 91 is used to store computer programs and other programs and data required by the electronic device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0129] It should be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiment, which will not be described here.

[0130] An embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the method embodiment of the real-time rendering point cloud.

[0131] An embodiment of the present application provides a computer program product, when the computer program product runs on an electronic device, causes the electronic device to implement the steps in the method embodiment of real-time rendering of a point cloud.

[0132] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0133] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0134] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0135] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0137] The integrated modules / units, if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electric carrier signals and telecommunication signals.

[0138] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of rendering a point cloud in real time, characterized in that, The point cloud is rendered using a mesh filter in the Unity engine, including: Point cloud data is obtained using data collected during laser radar scanning, and the point cloud data is initialized to obtain initialized point cloud data; The initialized point cloud data is loaded in parallel, the initialized point cloud data is converted into a point cloud array, and the Unity engine is called to convert the point cloud array into mesh vertices in parallel and transmit the mesh vertices to the GPU; obtaining the mesh vertices and transmitting the mesh vertices to the GPU is an asynchronous logical instruction, the asynchronous logical instruction is packaged into multiple individual elements, multiple point cloud arrays are read by multiple threads at the same time for parallel processing to obtain multiple mesh vertices and load multiple mesh vertices from the CPU to the GPU in parallel; The coordinates of each vertex in the mesh vertex are subjected to affine transformation to realize point cloud rendering, including stretching the coordinates of each vertex in the mesh vertex using the shader of the Unity engine in the GPU, calculating the new coordinates of each vertex after stretching, and rendering the point cloud onto the pixel screen according to the new coordinates of each vertex.

2. The method of rendering a point cloud in real-time of claim 1, wherein, The initialized point cloud data is loaded in parallel, and the Unity engine is called to convert the point cloud array into mesh vertices in parallel, including: The point cloud array is transmitted to the shared memory of the Unity engine; The point cloud array in the shared memory is processed in parallel to obtain the mesh vertices of the point cloud array, and the mesh vertices are transmitted to the GPU.

3. The method of rendering a point cloud in real-time of claim 1, wherein, The initialized point cloud data is loaded in parallel, and the Unity engine is called to convert the point cloud array into mesh vertices in parallel, including: After the initialized point cloud data is converted into a point cloud array, it is transmitted to the storage area of the Unity engine; The point cloud array in the storage area is processed in parallel to obtain the mesh vertices of the point cloud array, and the mesh vertices are transmitted to the GPU.

4. The method of rendering a point cloud in real time of any one of claims 1 to 3, wherein, The initialized point cloud data is loaded in parallel, and the Unity engine is called to convert the point cloud array into mesh vertices in parallel, including: The point cloud data is initialized in a predetermined format to obtain point cloud data in a predetermined format.

5. The method of rendering a point cloud in real time of any one of claims 1 to 3, wherein, Point cloud data is obtained using data collected during laser radar scanning, including: Depth images and their basic data collected during laser radar scanning are obtained, and point cloud data is obtained according to the depth images and their basic data.

6. An apparatus for rendering a point cloud in real time, the apparatus comprising: The point cloud is rendered using a mesh filter in the Unity engine, including: An initialization module is configured to obtain point cloud data using data collected during laser radar scanning, and to initialize the point cloud data to obtain initialized point cloud data; A conversion module is configured to load the initialized point cloud data in parallel, convert the initialized point cloud data into a point cloud array, and convert the point cloud array into mesh vertices in parallel and transmit the mesh vertices to the GPU; obtaining the mesh vertices and transmitting the mesh vertices to the GPU is an asynchronous logical instruction, the asynchronous logical instruction is packaged into multiple individual elements, multiple point cloud arrays are read by multiple threads at the same time for parallel processing to obtain multiple mesh vertices and load multiple mesh vertices from the CPU to the GPU in parallel; The rendering module is configured to perform affine transformation on the coordinates of each vertex in the grid vertices to realize point cloud rendering, including stretching the coordinates of each vertex in the grid vertices using a shader of the Unity engine in the GPU, and calculating new coordinates of each vertex after stretching, and rendering the point cloud onto a pixel screen according to the new coordinates of each vertex. The conversion module and the rendering module can call the Unity engine to work.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the method for rendering a point cloud in real time according to any one of claims 1 to 5.

8. A system for real-time rendering of point clouds, characterized in that The method comprises: One or more laser radars, and the electronic device according to claim 7, wherein the one or more laser radars are connected to the electronic device.

9. A computer storage medium storing a computer program, the computer program comprising instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 8. The computer program is executed by the processor to realize the method for rendering a point cloud in real time according to any one of claims 1 to 5.

Images