Point cloud sequence transmission method and device, computer equipment, storage medium and product

Abstract

The invention relates to a point cloud sequence transmission method and device, computer equipment, a storage medium and a product. The method comprises the following steps: acquiring an environment point cloud sequence of a driving environment in which a target vehicle is located; extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution condition of the environment point cloud sequence; performing semantic compression processing on the core point cloud sequence according to a point cloud data transmission requirement; wherein the semantic compression processing comprises semantic segmentation processing and / or space-time compression processing; and transmitting the core point cloud sequence after semantic compression processing. By adopting the method, the continuous point cloud sequence can be efficiently compressed and transmitted.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a point cloud sequence transmission method, device, computer equipment, storage medium and product. Background Art

[0002] LiDAR (Light Detection and Ranging) sensors play a vital role in the field of autonomous driving, providing vehicles with three-dimensional all-round perception of the surrounding environment. However, LiDAR point clouds generate a very large amount of data. For example, acquiring a 3D frame of 700,000 points at 30 frames per second requires a bandwidth of about 500 Mbps, which is challenging for standard communication systems. Therefore, real-time compression and transmission of a large number of LiDAR point clouds is crucial for autonomous vehicles.

[0003] Previous point cloud compression methods address this research challenge by projecting 3D point clouds into 2D representations for 2D compression, thus achieving real-time performance; however, these methods are inefficient and are only applied to a single point cloud frame. Summary of the invention

[0004] Based on this, it is necessary to provide a point cloud sequence transmission method, device, computer equipment, storage medium and product that can efficiently compress and transmit continuous point cloud sequences in order to address the above technical problems.

[0005] In a first aspect, the present application provides a point cloud sequence transmission method, comprising:

[0006] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0007] Extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence;

[0008] According to the point cloud data transmission requirements, the core point cloud sequence is subjected to semantic compression processing; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0009] The core point cloud sequence after semantic compression is transmitted.

[0010] In one embodiment, the semantic compression processing of the core point cloud sequence according to the point cloud data transmission requirements includes:

[0011] If the point cloud data transmission requirement is the first data transmission requirement, performing spatiotemporal compression processing on the core point cloud sequence;

[0012] If the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, performing semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence;

[0013] Among them, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0014] In one embodiment, the performing semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence includes:

[0015] According to the fully convolutional semantic segmentation network, the core point cloud sequence is subjected to semantic segmentation processing to determine the candidate semantic category corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic category includes a road category, a static object category, or a dynamic object category;

[0016] determining a target semantic category from the candidate semantic categories;

[0017] The point cloud corresponding to the target semantic category is subjected to spatiotemporal compression processing.

[0018] In one embodiment, determining a target semantic category from the candidate semantic categories comprises:

[0019] In a case where the point cloud data transmission requirement is the second data transmission requirement, a static object category and a dynamic object category are used as target semantic categories;

[0020] When the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is used as the target semantic category.

[0021] In one of the embodiments, the semantic segmentation processing is performed on the core point cloud sequence according to the fully convolutional semantic segmentation network to determine the candidate semantic category corresponding to each point cloud in the core point cloud sequence, including:

[0022] Converting the core point cloud sequence into a range image by a spherical projection method;

[0023] The distance image corresponding to the core point cloud sequence is input into a fully convolutional semantic segmentation network to obtain candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0024] In one embodiment, the performing spatiotemporal compression processing on the core point cloud sequence includes:

[0025] Dividing the core point cloud sequence to obtain key point cloud sequences and other point cloud sequences;

[0026] The key point cloud sequence is spatially encoded on a fitting plane, and the other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequence.

[0027] In a second aspect, the present application also provides a point cloud sequence transmission device, comprising:

[0028] A sequence acquisition module is used to acquire an environmental point cloud sequence of the driving environment of the target vehicle;

[0029] A sequence extraction module, used for extracting a core point cloud sequence from the environmental point cloud sequence according to the point cloud density distribution of the environmental point cloud sequence;

[0030] A semantic compression module, used for performing semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0031] The sequence transmission module is used to transmit the core point cloud sequence after semantic compression processing.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0034] Extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence;

[0035] According to the point cloud data transmission requirements, the core point cloud sequence is subjected to semantic compression processing; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0036] The core point cloud sequence after semantic compression is transmitted.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0039] Extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence;

[0040] According to the point cloud data transmission requirements, the core point cloud sequence is subjected to semantic compression processing; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0041] The core point cloud sequence after semantic compression is transmitted.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0043] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0044] Extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence;

[0045] According to the point cloud data transmission requirements, the core point cloud sequence is subjected to semantic compression processing; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0046] The core point cloud sequence after semantic compression is transmitted.

[0047] The above-mentioned point cloud sequence transmission method, device, computer equipment, storage medium and product obtain the environmental point cloud sequence of the driving environment of the target vehicle; extract the core point cloud sequence from the environmental point cloud sequence according to the point cloud density distribution of the environmental point cloud sequence; perform semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; and transmit the core point cloud sequence after semantic compression processing. The above-mentioned scheme reduces the data volume of the environmental point cloud sequence while ensuring the important data (i.e., the core point cloud sequence) in the environmental point cloud sequence by extracting the core point cloud sequence according to the point cloud density distribution of the environmental point cloud; at the same time, in combination with the point cloud data transmission requirements, different semantic compression processing (i.e., semantic segmentation processing and / or spatiotemporal compression processing) is performed on the core point cloud sequence, thereby improving the flexibility of the environmental point cloud sequence in compression and transmission, and by performing semantic compression processing on the core point cloud sequence, the efficiency of the environmental point cloud sequence in the transmission process is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1A diagram showing an application environment of a point cloud sequence transmission method in one embodiment;

[0050] Figure 2 Schematic diagram of a flow chart of a point cloud sequence transmission method in one embodiment;

[0051] Figure 3 A schematic diagram of a process flow for performing semantic segmentation processing and spatiotemporal compression processing in one embodiment;

[0052] Figure 4 A schematic diagram of a process for determining a candidate semantic category in one embodiment;

[0053] Figure 5 It is a schematic diagram of a process of performing spatiotemporal compression processing on a core point cloud sequence in one embodiment;

[0054] Figure 6 A schematic diagram of a flow chart of a point cloud sequence transmission method in another embodiment;

[0055] Figure 7 is a structural block diagram of a point cloud sequence transmission device in one embodiment;

[0056] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The point cloud sequence transmission method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the laser radar 101 is a radar device in the target vehicle for scanning the driving environment of the target vehicle. In the embodiment of the present application, the laser radar can be a LiDAR sensor; the automatic driving control system 102 is a system in the target vehicle for controlling the automatic driving of the target vehicle. Optionally, after the laser radar 101 obtains the environmental point cloud sequence of the driving environment of the target vehicle, it extracts the core point cloud sequence from the environmental point cloud sequence according to the point cloud density distribution of the environmental point cloud sequence; further, according to the point cloud data transmission requirements, the core point cloud sequence is semantically compressed; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; and the core point cloud sequence after the semantic compression processing is transmitted to the automatic driving control system 102.

[0059] In one embodiment, Figure 2 As shown, a point cloud sequence transmission method is provided, and the method is applied to Figure 1The laser radar 101 in the embodiment is used as an example to illustrate the invention, and specifically includes the following steps:

[0060] S201, obtaining an environmental point cloud sequence of the driving environment of the target vehicle.

[0061] The target vehicle may be any vehicle equipped with a laser radar; the environmental point cloud sequence is a point cloud sequence obtained by scanning the target vehicle's driving environment with the target vehicle's laser radar.

[0062] Optionally, when the target vehicle is driving, in order to ensure driving safety, a laser radar in the target vehicle, such as a laser radar based on a LiDAR sensor, will scan the driving environment of the target vehicle in real time and obtain an environmental point cloud sequence of the driving environment of the target vehicle.

[0063] S202: extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence.

[0064] The point cloud density distribution characterizes the number of point cloud points in a unit area, that is, it reflects the characteristics and density of the point cloud spatial distribution. The core point cloud sequence is a sequence of point clouds containing important information.

[0065] Optionally, a density-based clustering algorithm may be used to determine the point cloud density of each unit area in the environmental point cloud sequence; further, points within a given radius whose point cloud quantity is greater than or equal to a quantity threshold are taken as core point cloud sequences.

[0066] S203, according to the point cloud data transmission requirements, semantic compression processing is performed on the core point cloud sequence.

[0067] The point cloud data transmission requirement is the requirement for the data transmission accuracy of the point cloud to be transmitted; in the embodiment of the present application, the data transmission accuracy can be reflected in the size of the memory occupied by the point cloud sequence to be transmitted, the importance of the content contained in the point cloud sequence to be transmitted, etc. The semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing.

[0068] It is understandable that the complexity of the road conditions during peak and off-peak traffic periods is different, which means that the amount of core point cloud data collected by the lidar is also different. On this basis, in order to ensure the real-time and integrity of the core point cloud sequence transmission, different point cloud data transmission requirements can be formulated for different time periods.

[0069] Furthermore, different semantic compression processing can be performed on the core point cloud data according to different point cloud data transmission requirements. For example, if the point cloud data transmission requires low data transmission accuracy, only the core point cloud sequence can be subjected to spatiotemporal compression processing. In this case, the core point cloud sequence to be transmitted occupies a large amount of memory and contains a lot of unimportant information. If the point cloud data transmission requires high data transmission accuracy, it is necessary to first perform semantic segmentation processing on the core point cloud sequence to extract relatively important data and reduce the memory of the core point cloud sequence. And the extracted important point cloud sequence is subjected to spatiotemporal compression processing.

[0070] S204, transmitting the core point cloud sequence after the semantic compression processing.

[0071] Optionally, the core point cloud sequence after semantic compression processing is transmitted through a communication channel.

[0072] In the above point cloud sequence transmission method, the environment point cloud sequence of the target vehicle's driving environment is obtained; according to the point cloud density distribution of the environment point cloud sequence, the core point cloud sequence is extracted from the environment point cloud sequence; according to the point cloud data transmission requirements, the core point cloud sequence is semantically compressed; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; the core point cloud sequence after the semantic compression processing is transmitted. The above scheme reduces the data volume of the environment point cloud sequence while ensuring the important data (i.e., the core point cloud sequence) in the environment point cloud sequence by extracting the core point cloud sequence according to the point cloud density distribution of the environment point cloud; at the same time, in combination with the point cloud data transmission requirements, different semantic compression processing (i.e., semantic segmentation processing and / or spatiotemporal compression processing) is performed on the core point cloud sequence, thereby improving the flexibility of the environment point cloud sequence in compression and transmission, and by performing semantic compression processing on the core point cloud sequence, the efficiency of the environment point cloud sequence in the transmission process is also improved.

[0073] Optionally, based on the above embodiment, in one embodiment, the semantic compression processing process of the core point cloud sequence according to the point cloud data transmission requirements described in S203 is refined. Optionally, the point cloud data transmission requirements are divided into a first data transmission requirement, a second data transmission requirement, and a third data transmission requirement, and the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0074] On this basis, if the point cloud data transmission requirement is the first data transmission requirement, the core point cloud sequence is subjected to spatiotemporal compression. Specifically, if the point cloud data transmission requirement is the first data transmission requirement, it means that the data transmission accuracy required for the core point cloud sequence is low. At this point, it is only necessary to perform spatiotemporal compression on the core point cloud sequence. However, if the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, it means that the core point cloud sequence needs to be further compressed to reduce the memory size of the core point cloud sequence. At this point, the core point cloud sequence can be semantically segmented first, and then the semantically segmented core point cloud sequence can be spatiotemporally compressed.

[0075] Optionally, in order to ensure that when the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, the core point cloud sequence after semantic compression processing better meets the requirement, in one embodiment, Figure 3 As shown, a method for semantic segmentation and spatiotemporal compression of a core point cloud sequence is provided, which specifically includes the following steps:

[0076] S301, performing semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network, and determining the candidate semantic category corresponding to each point cloud in the core point cloud sequence.

[0077] Among them, the fully convolutional semantic segmentation network is an end-to-end fully convolutional network for pixel-level prediction, which is used to perform semantic segmentation on the core point cloud sequence; the candidate semantic categories include road categories, object static object categories or object dynamic object categories; the road category includes point clouds related to the road on which the target vehicle is traveling; the object static object category includes point clouds related to static objects in the environment in which the target vehicle is traveling, such as point clouds related to static objects such as buildings, vegetation and traffic signs; the object dynamic object category includes point clouds related to dynamic objects in the environment in which the target vehicle is traveling, such as point clouds related to dynamic objects such as pedestrians and vehicles.

[0078] Optionally, the core point cloud sequence can be input into a fully convolutional semantic segmentation network, so that the fully convolutional semantic segmentation network performs semantic segmentation on the core point cloud sequence according to network parameters, and outputs candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0079] S302: Determine a target semantic category from candidate semantic categories.

[0080] The target semantic category is a semantic category corresponding to the point cloud that needs to be transmitted, and the number of target semantic categories can be multiple.

[0081] Optionally, the most important candidate semantic category may be determined from among the candidate semantic categories according to the importance of each candidate semantic category as the target semantic category.

[0082] It is understandable that in order to ensure that the determined target semantic category can meet the point cloud data transmission requirements, the target semantic category can also be selected from the candidate semantic categories in combination with the point cloud data transmission requirements. Optionally, when the point cloud data transmission requirement is the second data transmission requirement, the static object category and the dynamic object category are used as the target semantic category. Specifically, considering that the road type will not change within a certain period of time during the vehicle's transformation process, therefore, when the point cloud data transmission requirement is the second data transmission requirement, it is only necessary to use the static object category and the dynamic object category as the target semantic category.

[0083] Optionally, when the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is used as the target semantic category. Specifically, since the data transmission accuracy corresponding to the third data transmission requirement is very high, the most important dynamic object category can be used as the target semantic category to achieve the purpose of compressing the core point cloud sequence.

[0084] S303, performing spatiotemporal compression processing on the point cloud corresponding to the target semantic category.

[0085] Optionally, the temporal and spatial redundancy of the point cloud corresponding to the target semantic category may be utilized, and a spatiotemporal compression method may be adopted to perform spatiotemporal compression processing on the point cloud corresponding to the target semantic category.

[0086] In this embodiment, by introducing candidate semantic categories and extracting target semantic categories from the candidate semantic categories, the memory of the point cloud sequence that needs to be transmitted is reduced to a certain extent, the efficiency of the spatiotemporal compression processing is ensured, and the transmission efficiency of the transmission process is further ensured.

[0087] Optionally, in one embodiment, Figure 4 As shown, a method for determining a candidate semantic category is provided, which specifically includes the following steps:

[0088] S401, converting the core point cloud sequence into a distance image through a spherical projection method.

[0089] Among them, the distance image represents the planar distance between the core point cloud sequence and the corresponding actual object.

[0090] Optionally, each point cloud in the three-dimensional core point cloud sequence is projected onto a plane by a spherical projection method, so as to convert the core point cloud sequence into a range image.

[0091] S402, inputting the distance image corresponding to the core point cloud sequence into the fully convolutional semantic segmentation network to obtain the candidate semantic category corresponding to each point cloud in the core point cloud sequence.

[0092] Optionally, the distance image corresponding to the core point cloud sequence can be input into a fully convolutional semantic segmentation network, so that the fully convolutional semantic segmentation network performs semantic segmentation on the distance image and outputs candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0093] In the embodiment of the present application, by converting the core point cloud sequence into a distance image, the accuracy of the fully convolutional semantic segmentation network is guaranteed, and the accuracy of the candidate semantic categories corresponding to each determined point cloud is guaranteed.

[0094] Optionally, in one embodiment, Figure 5 As shown, a method for performing spatiotemporal compression processing on a core point cloud sequence is provided, which specifically includes the following steps:

[0095] S501, dividing the core point cloud sequence to obtain key point cloud sequences and other point cloud sequences.

[0096] Among them, the point clouds in the key point cloud sequence represent the important elements in the core point cloud sequence, and generally include the intermediate point clouds in the core point cloud sequence. Other point cloud sequences include point clouds in the core point cloud sequence except for the point clouds in the key point cloud sequence.

[0097] Optionally, a key point cloud sequence is extracted from the core point cloud sequence by using a point cloud extraction tool, and point cloud sequences other than the key point cloud sequence in the core point cloud sequence are used as other point cloud sequences.

[0098] S502, performing spatial encoding of the fitting plane on the key point cloud sequence, and performing temporal encoding on other point cloud sequences, so as to achieve spatiotemporal compression processing on the core point cloud sequence.

[0099] Optionally, the key point cloud sequence in the sequence is spatially encoded by the fitting plane, and then the spatial encoding result is used to temporally encode other point cloud sequences to achieve spatiotemporal compression processing of the core point cloud sequence.

[0100] In the embodiment of the present application, the core point cloud sequence is divided into a key point cloud sequence and other point cloud sequences, and the key point cloud sequence and the other point cloud sequences are spatially encoded and temporally encoded respectively, thereby accelerating the compression speed while retaining key information.

[0101] Figure 6 FIG. 2 is a flow chart of a method for transmitting a point cloud sequence in another embodiment. Based on the above embodiment, this embodiment provides an optional example of a method for transmitting a point cloud sequence. Figure 6 The specific implementation process is as follows:

[0102] S601, obtaining an environmental point cloud sequence of the driving environment of the target vehicle.

[0103] S602: extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence.

[0104] S603, determining whether the point cloud data transmission requirement is the first data transmission requirement; if so, executing S604; if not, executing S605.

[0105] S604, performing spatiotemporal compression processing on the core point cloud sequence, and executing S610.

[0106] Optionally, the core point cloud sequence is divided to obtain key point cloud sequences and other point cloud sequences; the key point cloud sequences are spatially encoded on the fitting plane, and other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequences.

[0107] S605, performing semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network, and determining the candidate semantic category corresponding to each point cloud in the core point cloud sequence.

[0108] The candidate semantic categories include road categories, static object categories, or dynamic object categories.

[0109] Optionally, the core point cloud sequence is converted into a distance image by a spherical projection method; the distance image corresponding to the core point cloud sequence is input into a fully convolutional semantic segmentation network to obtain candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0110] S606, determining whether the point cloud data transmission requirement is the second data transmission requirement; if so, executing S607; if not, executing S608.

[0111] S607 , taking the static object category and the dynamic object category as the target semantic category, and executing S609 .

[0112] S608, taking the dynamic object category as the target semantic category, and executing S609.

[0113] S609: performing spatiotemporal compression processing on the point cloud corresponding to the target semantic category.

[0114] S610, transmitting the core point cloud sequence after the spatiotemporal compression processing, or the point cloud corresponding to the target semantic category after the spatiotemporal compression processing.

[0115] The specific process of S601-S610 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0116] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0117] Based on the same inventive concept, the embodiment of the present application also provides a point cloud sequence transmission device for implementing the point cloud sequence transmission method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more point cloud sequence transmission device embodiments provided below can refer to the above limitations on the point cloud sequence transmission method, and will not be repeated here.

[0118] In one embodiment, Figure 7 As shown, a point cloud sequence transmission device 700 is provided, comprising: a sequence acquisition module 710, a sequence extraction module 720, a semantic compression module 730 and a sequence transmission module 740, wherein:

[0119] The sequence acquisition module 710 is used to acquire the environmental point cloud sequence of the driving environment of the target vehicle.

[0120] The sequence extraction module 720 is used to extract a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence.

[0121] The semantic compression module 730 is used to perform semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing.

[0122] The sequence transmission module 740 is used to transmit the core point cloud sequence after the semantic compression processing.

[0123] The above-mentioned point cloud sequence transmission device obtains the environmental point cloud sequence of the driving environment where the target vehicle is located; extracts the core point cloud sequence from the environmental point cloud sequence according to the point cloud density distribution of the environmental point cloud sequence; performs semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; and transmits the core point cloud sequence after the semantic compression processing. The above-mentioned scheme reduces the data volume of the environmental point cloud sequence while ensuring the important data (i.e., the core point cloud sequence) in the environmental point cloud sequence by extracting the core point cloud sequence according to the point cloud density distribution of the environmental point cloud; at the same time, in combination with the point cloud data transmission requirements, performs different semantic compression processing (i.e., semantic segmentation processing and / or spatiotemporal compression processing) on ​​the core point cloud sequence, thereby improving the flexibility of the environmental point cloud sequence in compression and transmission, and also improves the efficiency of the environmental point cloud sequence in the transmission process by performing semantic compression processing on the core point cloud sequence.

[0124] In one embodiment, the semantic compression module 730 includes:

[0125] The first compression unit is used to perform spatiotemporal compression processing on the core point cloud sequence if the point cloud data transmission requirement is the first data transmission requirement.

[0126] The second compression unit performs semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence if the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement.

[0127] Among them, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0128] In one embodiment, the second compression unit comprises:

[0129] The segmentation subunit is used to perform semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network, and determine the candidate semantic categories corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic categories include road categories, static object categories or dynamic object categories.

[0130] A determination subunit is used to determine a target semantic category from candidate semantic categories.

[0131] The compression subunit is used to perform spatiotemporal compression processing on the point cloud corresponding to the target semantic category.

[0132] In one embodiment, the determining subunit is configured to:

[0133] When the point cloud data transmission requirement is the second data transmission requirement, the static object category and the dynamic object category are taken as the target semantic category; when the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is taken as the target semantic category.

[0134] In one embodiment, the segmentation subunit is used to:

[0135] The core point cloud sequence is converted into a distance image through the spherical projection method. The distance image corresponding to the core point cloud sequence is input into the fully convolutional semantic segmentation network to obtain the candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0136] In one embodiment, the first compression unit is specifically configured to:

[0137] The core point cloud sequence is divided to obtain key point cloud sequences and other point cloud sequences; the key point cloud sequences are spatially encoded by fitting planes, and other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequences.

[0138] Each module in the above-mentioned point cloud sequence transmission device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a point cloud sequence transmission method is implemented.

[0140] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0142] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0143] According to the point cloud density distribution of the environment point cloud sequence, a core point cloud sequence is extracted from the environment point cloud sequence;

[0144] According to the point cloud data transmission requirements, the core point cloud sequence is semantically compressed; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0145] The core point cloud sequence after semantic compression is transmitted.

[0146] In one embodiment, when the processor executes the computer program to perform semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements, the processor also implements the following steps:

[0147] If the point cloud data transmission requirement is the first data transmission requirement, the core point cloud sequence is subjected to spatiotemporal compression processing; if the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, the core point cloud sequence is subjected to semantic segmentation processing and spatiotemporal compression processing; wherein, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; and the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0148] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0149] According to the fully convolutional semantic segmentation network, the core point cloud sequence is semantically segmented to determine the candidate semantic categories corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic categories include road categories, static object categories or dynamic object categories; the target semantic category is determined from the candidate semantic categories; and the point cloud corresponding to the target semantic category is subjected to spatiotemporal compression processing.

[0150] In one embodiment, when the processor executes the computer program to determine the target semantic category from the candidate semantic categories, the processor further implements the following steps:

[0151] When the point cloud data transmission requirement is the second data transmission requirement, the static object category and the dynamic object category are taken as the target semantic category; when the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is taken as the target semantic category.

[0152] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network and determines the candidate semantic category corresponding to each point cloud in the core point cloud sequence, the following steps are also implemented:

[0153] The core point cloud sequence is converted into a distance image through the spherical projection method. The distance image corresponding to the core point cloud sequence is input into the fully convolutional semantic segmentation network to obtain the candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0154] In one embodiment, when the processor executes the computer program to perform spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0155] The core point cloud sequence is divided to obtain key point cloud sequences and other point cloud sequences; the key point cloud sequences are spatially encoded by fitting planes, and other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequences.

[0156] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0157] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0158] According to the point cloud density distribution of the environment point cloud sequence, a core point cloud sequence is extracted from the environment point cloud sequence;

[0159] According to the point cloud data transmission requirements, the core point cloud sequence is semantically compressed; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0160] The core point cloud sequence after semantic compression is transmitted.

[0161] In one embodiment, when the processor executes the computer program to perform semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements, the processor also implements the following steps:

[0162] If the point cloud data transmission requirement is the first data transmission requirement, the core point cloud sequence is subjected to spatiotemporal compression processing; if the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, the core point cloud sequence is subjected to semantic segmentation processing and spatiotemporal compression processing; wherein, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; and the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0163] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0164] According to the fully convolutional semantic segmentation network, the core point cloud sequence is semantically segmented to determine the candidate semantic categories corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic categories include road categories, static object categories or dynamic object categories; the target semantic category is determined from the candidate semantic categories; and the point cloud corresponding to the target semantic category is subjected to spatiotemporal compression processing.

[0165] In one embodiment, when the processor executes the computer program to determine the target semantic category from the candidate semantic categories, the processor further implements the following steps:

[0166] When the point cloud data transmission requirement is the second data transmission requirement, the static object category and the dynamic object category are taken as the target semantic category; when the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is taken as the target semantic category.

[0167] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network and determines the candidate semantic category corresponding to each point cloud in the core point cloud sequence, the following steps are also implemented:

[0168] The core point cloud sequence is converted into a distance image through the spherical projection method. The distance image corresponding to the core point cloud sequence is input into the fully convolutional semantic segmentation network to obtain the candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0169] In one embodiment, when the processor executes the computer program to perform spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0170] The core point cloud sequence is divided to obtain key point cloud sequences and other point cloud sequences; the key point cloud sequences are spatially encoded by fitting planes, and other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequences.

[0171] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0172] Obtaining an environmental point cloud sequence of the target vehicle's driving environment;

[0173] According to the point cloud density distribution of the environment point cloud sequence, a core point cloud sequence is extracted from the environment point cloud sequence;

[0174] According to the point cloud data transmission requirements, the core point cloud sequence is semantically compressed; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing;

[0175] The core point cloud sequence after semantic compression is transmitted.

[0176] In one embodiment, when the processor executes the computer program to perform semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements, the processor also implements the following steps:

[0177] If the point cloud data transmission requirement is the first data transmission requirement, the core point cloud sequence is subjected to spatiotemporal compression processing; if the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, the core point cloud sequence is subjected to semantic segmentation processing and spatiotemporal compression processing; wherein, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; and the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

[0178] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0179] According to the fully convolutional semantic segmentation network, the core point cloud sequence is semantically segmented to determine the candidate semantic categories corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic categories include road categories, static object categories or dynamic object categories; the target semantic category is determined from the candidate semantic categories; and the point cloud corresponding to the target semantic category is subjected to spatiotemporal compression processing.

[0180] In one embodiment, when the processor executes the computer program to determine the target semantic category from the candidate semantic categories, the processor further implements the following steps:

[0181] When the point cloud data transmission requirement is the second data transmission requirement, the static object category and the dynamic object category are taken as the target semantic category; when the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is taken as the target semantic category.

[0182] In one embodiment, when the processor executes the computer program to perform semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network and determines the candidate semantic category corresponding to each point cloud in the core point cloud sequence, the following steps are also implemented:

[0183] The core point cloud sequence is converted into a distance image through the spherical projection method. The distance image corresponding to the core point cloud sequence is input into the fully convolutional semantic segmentation network to obtain the candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

[0184] In one embodiment, when the processor executes the computer program to perform spatiotemporal compression processing on the core point cloud sequence, the processor further implements the following steps:

[0185] The core point cloud sequence is divided to obtain key point cloud sequences and other point cloud sequences; the key point cloud sequences are spatially encoded by fitting planes, and other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequences.

[0186] It should be noted that the data involved in this application (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0188] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A point cloud sequence transmission method, characterized in that: The method comprises: Obtaining an environmental point cloud sequence of the target vehicle's driving environment; Extracting a core point cloud sequence from the environment point cloud sequence according to the point cloud density distribution of the environment point cloud sequence; According to the point cloud data transmission requirements, the core point cloud sequence is subjected to semantic compression processing; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; The core point cloud sequence after semantic compression is transmitted.

2. The method according to claim 1, characterized in that The performing semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements includes: If the point cloud data transmission requirement is the first data transmission requirement, performing spatiotemporal compression processing on the core point cloud sequence; If the point cloud data transmission requirement is the second data transmission requirement or the third data transmission requirement, performing semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence; Among them, the data transmission accuracy corresponding to the first data transmission requirement is lower than the data transmission accuracy corresponding to the second data transmission requirement; the data transmission accuracy corresponding to the second data transmission requirement is lower than the data transmission accuracy corresponding to the third data transmission requirement.

3. The method according to claim 2, characterized in that The performing semantic segmentation processing and spatiotemporal compression processing on the core point cloud sequence includes: According to the fully convolutional semantic segmentation network, the core point cloud sequence is subjected to semantic segmentation processing to determine the candidate semantic category corresponding to each point cloud in the core point cloud sequence; wherein the candidate semantic category includes a road category, a static object category, or a dynamic object category; determining a target semantic category from the candidate semantic categories; The point cloud corresponding to the target semantic category is subjected to spatiotemporal compression processing.

4. The method according to claim 3, characterized in that The step of determining a target semantic category from the candidate semantic categories comprises: In a case where the point cloud data transmission requirement is the second data transmission requirement, a static object category and a dynamic object category are used as target semantic categories; When the point cloud data transmission requirement is the third data transmission requirement, the dynamic object category is used as the target semantic category.

5. The method according to claim 3, characterized in that: The step of performing semantic segmentation processing on the core point cloud sequence according to the fully convolutional semantic segmentation network to determine the candidate semantic category corresponding to each point cloud in the core point cloud sequence includes: Converting the core point cloud sequence into a range image by a spherical projection method; The distance image corresponding to the core point cloud sequence is input into a fully convolutional semantic segmentation network to obtain candidate semantic categories corresponding to each point cloud in the core point cloud sequence.

6. The method according to claim 2, characterized in that The performing spatiotemporal compression processing on the core point cloud sequence includes: Dividing the core point cloud sequence to obtain key point cloud sequences and other point cloud sequences; The key point cloud sequence is spatially encoded on a fitting plane, and the other point cloud sequences are temporally encoded to achieve spatiotemporal compression processing of the core point cloud sequence.

7. A point cloud sequence transmission device, characterized in that: The device comprises: A sequence acquisition module is used to acquire an environmental point cloud sequence of the driving environment of the target vehicle; A sequence extraction module, used for extracting a core point cloud sequence from the environmental point cloud sequence according to the point cloud density distribution of the environmental point cloud sequence; A semantic compression module, used for performing semantic compression processing on the core point cloud sequence according to the point cloud data transmission requirements; wherein the semantic compression processing includes semantic segmentation processing and / or spatiotemporal compression processing; The sequence transmission module is used to transmit the core point cloud sequence after semantic compression processing.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images