Autonomous driving scene perception method, system, and non-transitory computer readable medium

Abstract

The automatic driving scene perception method and system provided in the specification obtains target point cloud data of a target scene, inputs the target point cloud data into a target neural network comprising a backbone network and multiple branch networks, and outputs perception results of multiple tasks respectively, that is, multiple tasks are coupled into the same neural network framework for processing, different tasks share the same backbone network and the same feature information, multiple perception tasks are simultaneously implemented, which is conducive to improving the real-time performance and performance of the network, and multiple scene perception requirements required by automatic driving can be implemented in the same framework.

Description

Technical Field

[0001] This specification relates to the field of deep learning technology, and in particular to an autonomous driving scene perception method, system, and non-temporary computer-readable medium. Background Technology

[0002] Autonomous vehicles are self-driving transportation tools that integrate perception, decision-making, and control functions. The perception system replaces the human driver's visual, auditory, and tactile functions, integrating massive amounts of traffic environment data collected from multiple sensors to accurately identify various traffic elements and support the autonomous vehicle's decision-making system. LiDAR is one of the key sensors in autonomous vehicles, providing a reliable data acquisition method for point cloud-based autonomous driving scene perception methods.

[0003] Scene perception plays a crucial role in autonomous driving. LiDAR acquires point cloud data from the external environment and transmits it to the processor; software algorithms running on the processor perceive pedestrians, vehicles, and drivable areas. Traditional perception algorithms fail to achieve the expected accuracy and real-time performance in handling complex traffic scenarios. While deep learning-based convolutional neural network algorithms can train ideal models using massive training datasets, achieving environmental perception for autonomous driving requires simultaneously addressing multiple functions such as object detection, lane detection, traffic sign detection, drivable area segmentation, and curb segmentation. The limited computing power of onboard chips and the massive computational consumption of deep learning models become the biggest obstacle to practical application. Therefore, as autonomous driving technology research deepens, point cloud-based scene perception solutions are emerging as a key solution for the practical application of autonomous driving technology. Summary of the Invention

[0004] The autonomous driving scene perception method and system provided in this specification offer a target neural network based on point cloud data, which couples multiple tasks into the same neural network framework for processing. Different tasks share the same backbone network and the same feature information, enabling the realization of multiple scene perception requirements for autonomous driving within the same framework.

[0005] In a first aspect, this specification provides a target perception method, comprising: acquiring target point cloud data of a target scene; and inputting the target point cloud data into a target neural network to output perception results for multiple tasks, wherein the target neural network comprises at least: a backbone network configured to extract shared features from the target point cloud data and output a shared feature map, and multiple branch networks, each branch network being configured to output perception results for the multiple tasks based on the shared feature map.

[0006] In some embodiments, acquiring target point cloud data of a target scene includes: acquiring three-dimensional raw point cloud data obtained by lidar scanning, preprocessing the three-dimensional raw point cloud data to obtain first projection map data, and determining the first projection map data as the target point cloud data.

[0007] In some embodiments, the backbone network comprises a shared network with K levels of cascaded connections, where K is an integer greater than or equal to 1.

[0008] In some embodiments, inputting the target point cloud data into the target neural network includes: inputting the target point cloud data into the K-level shared network for feature extraction, and obtaining the i-th level shared feature map of the target point cloud data at the output of the i-th level shared network, where 1≥i≥K.

[0009] In some embodiments, inputting the target point cloud data into the target neural network further includes: inputting the target point cloud data into the first-level shared network of the K-level shared network for feature extraction to obtain a first-level shared feature map, wherein the first-level shared feature map includes a feature map of the original point cloud data from a first perspective; and based on the first-level shared feature map and the second-level shared network of the K-level shared network, obtaining a second-level shared feature map from the output of the second-level shared network, wherein the second-level shared feature map includes a feature map of the original point cloud data from a second perspective.

[0010] In some embodiments, the original point cloud data in the first viewpoint includes: front view data obtained by projecting the original point cloud data onto a cylindrical surface.

[0011] In some embodiments, the plurality of branch networks include a first type of branch network, the first type of branch network includes at least one branch network, and the output of perception results for each of the plurality of tasks includes: inputting the first-level shared feature map into the first type of branch network for perception, obtaining at least one perception result output by the first type of branch network, the first type of branch network performing perception based on the original point cloud data under the first viewpoint, wherein the first type of branch network processes at least one of the following tasks: semantic segmentation, lane line detection, and traffic sign detection, and the at least one perception result includes at least one of the following: semantic segmentation result, lane line detection result, and traffic sign detection result.

[0012] In some embodiments, the raw point cloud data in the second perspective includes: bird's-eye view data obtained by projecting the raw point cloud data onto a horizontal plane.

[0013] In some embodiments, obtaining the second-level shared feature map from the output of the second-level shared network based on the first-level shared feature map and the second-level shared network in the K-level shared network includes: performing a splicing and projection process on the first-level shared feature map and the target point cloud data to obtain second projection map data; and inputting the second projection map data into the second-level shared network for feature extraction to obtain the second-level shared feature map.

[0014] In some embodiments, the plurality of branch networks include a second type of branch network, the second type of branch network including at least one branch network, and the output of perception results for each of the plurality of tasks includes: inputting the second-level shared feature map into the second type of branch network for perception, obtaining at least one perception result output by the second type of branch network, the second type of branch network performing perception based on the original point cloud data under the second viewpoint, wherein the second type of branch network processes at least one of the following tasks: object detection, roadside detection, and drivable area segmentation, and the at least one perception result includes at least one of the following: object detection result, roadside detection result, and drivable area segmentation result.

[0015] In some embodiments, the target neural network is a network obtained by iterative training using a joint loss function, which is obtained by linearly weighting multiple loss functions of the multiple branch networks.

[0016] Secondly, this specification also provides an autonomous driving scene perception system, including at least one storage medium storing at least one set of instructions for autonomous driving scene perception; and at least one processor communicatively connected to the at least one storage medium, wherein when the autonomous driving scene perception system is running, the at least one processor reads the at least one set of instructions and implements the autonomous driving scene perception method described in the first aspect.

[0017] Thirdly, this specification also provides a non-temporary computer-readable medium storing at least one set of instructions for autonomous driving scene perception, wherein when the at least one set of instructions is executed by a processor, it instructs the processor to perform the autonomous driving scene perception method described in the first aspect.

[0018] As can be seen from the above technical solutions, the autonomous driving scene perception method and system provided in this specification offer a target neural network based on point cloud data. This network couples multiple tasks into the same neural network framework for processing. Different tasks share the same backbone network and the same feature information, allowing multiple perception tasks to be implemented simultaneously. This improves the network's real-time performance and allows for the fulfillment of various scene perception requirements for autonomous driving within a single framework. Furthermore, coupling multiple tasks into the same target neural network framework not only reduces computational load but also fully utilizes information from different tasks for training and iteration. Different tasks and functions are interconnected and mutually reinforcing.

[0019] The autonomous driving scene perception method and system provided in this specification adopt feature maps from different perspectives for different tasks. For example, it uses the first-level shared feature map in the first perspective to process tasks such as semantic segmentation, lane line detection, and traffic sign detection, and uses the second-level shared feature map in the second perspective to process tasks such as target detection, road edge detection, and drivable area segmentation. By making full use of the advantages of feature maps from different perspectives, a multi-view, multi-task autonomous driving scene perception method is constructed.

[0020] The autonomous driving scene perception method and system provided in this specification use a joint loss function composed of multiple branch networks to iteratively train the entire autonomous driving scene perception framework. While achieving multiple functions, the correlation of parameters between multiple tasks helps to better train the branch networks of each task.

[0021] The autonomous driving scene perception method and system provided in this specification utilize point cloud data, which has richer 3D information than images. The point cloud-based multi-task scene perception method for autonomous driving is more reliable than image algorithms and can achieve multiple perception requirements for autonomous driving within the same target neural network framework.

[0022] Other functions of the autonomous driving scene perception methods and systems provided in this specification will be partially listed in the following description. The figures and examples described below will be readily apparent to those skilled in the art. The inventive aspects of the object detection methods and systems provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A system architecture diagram 100 is shown for an autonomous driving scene perception method and system provided according to some embodiments of this specification;

[0025] Figure 2A A flowchart of an autonomous driving scene perception method P100 provided according to some embodiments of this specification is shown;

[0026] Figure 2B A schematic diagram of a multi-level shared backbone network provided according to some embodiments of this specification is shown;

[0027] Figure 3 A schematic diagram of raw point cloud data provided according to some embodiments of this specification is shown;

[0028] Figure 4 A schematic diagram of a front view data provided according to some embodiments of this specification is shown;

[0029] Figure 5 A framework diagram illustrating scene perception for autonomous driving via a Level 2 shared network, according to some embodiments of this specification, is shown.

[0030] Figure 6 A schematic diagram illustrating semantic segmentation results provided according to some embodiments of this specification is shown;

[0031] Figure 7 A schematic diagram showing lane lines in a driving scenario provided according to some embodiments of this specification is shown;

[0032] Figure 8 This diagram illustrates lane line detection results containing only lane lines, provided according to some embodiments of this specification.

[0033] Figure 9 This specification illustrates schematic diagrams showing traffic sign detection results according to some embodiments.

[0034] Figure 10 The diagram illustrates target detection results provided according to some embodiments of this specification;

[0035] Figure 11 This document shows schematic diagrams of curb detection results provided according to some embodiments of this specification;

[0036] Figure 12 A schematic diagram of drivable area segmentation provided according to some embodiments of this specification is shown;

[0037] Figure 13A framework diagram of semantic segmentation and object detection based on target point cloud data, according to some embodiments of this specification, is shown.

[0038] Figure 14 A schematic diagram of a U-net structure provided according to some embodiments of this specification is shown;

[0039] Figure 15 A schematic diagram of an FPN structure according to some embodiments of this specification is shown; and

[0040] Figure 16 A flowchart of a method P200 for training a target neural network in an autonomous driving scenario, according to some embodiments of this specification, is shown. Detailed Implementation

[0041] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0042] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0043] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0044] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0045] Before describing the specific embodiments in this specification, the application scenarios of this specification will be introduced as follows:

[0046] The autonomous driving scene perception methods and systems provided in this manual can be applied in autonomous driving scenarios. Achieving scene perception for autonomous driving requires simultaneously solving multiple tasks such as object detection, lane detection, traffic sign detection, drivable area segmentation, and curb detection. Building a separate backbone network for each task requires enormous memory and computing power. Therefore, building a shared backbone network within a multi-task neural network (object neural network) to extract features from data acquired by the corresponding sensors and accurately and efficiently identify all elements in the scene before feeding them into downstream processing is a crucial step.

[0047] As one of the important sensors for autonomous vehicles, LiDAR maintains a keen perception of the outside world by receiving feedback in real time. It has advantages such as high resolution, strong resistance to active interference, good directionality, long measurement distance, and short measurement time, and is less affected by the external environment than vision sensors.

[0048] For ease of description, this manual explains the terms that will appear in the following descriptions:

[0049] Point cloud data: Point cloud data collected by LiDAR, where each point includes its three-dimensional coordinates and reflectivity (x, y, z, r) in the LiDAR coordinate system. For ease of description, "point cloud data" can be referred to as "point cloud" or "point".

[0050] Figure 1 Figure 100 illustrates a system architecture diagram of an autonomous driving scene perception method and system according to some embodiments of this specification. The autonomous driving scene perception method and system provided in this specification can be applied to, for example... Figure 1 The system architecture 100 shown is as follows. (For example...) Figure 1 As shown, the system architecture 100 may include a data acquisition device 101 and an autonomous driving scene perception system 102. The data acquisition device 101 and the autonomous driving scene perception system 102 may be installed as a whole or installed separately.

[0051] The data acquisition device 101 can collect point cloud data in an autonomous driving scenario, manually or automatically label a portion of the point cloud data to obtain a training set, and use the remaining point cloud data as a test set. Then, the training set and test set are uploaded to the autonomous driving scenario perception system 102. The data acquisition device 101 can also collect target point cloud data of a target scene and send it to the autonomous driving scenario perception system 102. The data acquisition device 101 can also collect raw 3D point cloud data and send it to the autonomous driving scenario perception system 102, which preprocesses the raw cloud data to obtain a first projection map. Of course, the data acquisition device 101 can also collect other data besides point cloud data, such as data required for ground height regression. The data acquisition device 101 can be, for example, a mechanical lidar, a solid-state lidar, or a ground sensor.

[0052] The autonomous driving scene perception system 102 can execute the autonomous driving scene perception method described in this specification, which will be described in detail below. The autonomous driving scene perception system 102 is an exemplary computing device capable of executing the autonomous driving scene perception method. In some embodiments, the computing device may be a distributed computing device, including multiple distributed computing nodes.

[0053] The autonomous driving scene perception system 102 may include at least one processor 120 and at least one storage medium 130, wherein the at least one processor 120 may be communicatively connected to the at least one storage medium 130. In some embodiments, the autonomous driving scene perception system 102 may further include a communication module 150 and an internal communication bus 110. The at least one storage medium 130 stores the aforementioned training set, test set, and target point cloud data. When the autonomous driving scene perception system 102 is running, the at least one processor 120 may read the training set and test set, train a convolutional neural network using the training set, and test the trained convolutional neural network using the test set. After completing the training and testing, a target neural network whose performance meets preset indicators is obtained, and the target neural network is stored in the at least one storage medium 130. Thus, the at least one storage medium 130 also stores the target point cloud data and the trained target perception network. When the autonomous driving scene perception system 102 is running, the at least one processor 120 may read the target point cloud data and the target neural network, and execute the autonomous driving scene perception method provided in this specification.

[0054] At least one storage medium 130 may store the training set, test set, target point cloud data of the target scene, and target neural network in the form of an instruction set, which may be a collection of one or more instructions. For example, the instruction set may be multiple lines of computer program code, which, when retrieved and executed by the processor 120, may instruct the processor 120 to execute programs, routines, objects, components, data structures, processes, modules, etc., of the autonomous driving scene perception method provided in this specification. At least one storage medium 130 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 132, a read-only storage medium (ROM) 134, or a random access storage medium (RAM) 136.

[0055] At least one processor 120 can perform all the steps included in the autonomous driving scene perception method. The processor 120 can be in the form of one or more processors. In some embodiments, the processor 120 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof. For illustrative purposes only, only one processor 120 is described in the autonomous driving scene perception system 102 in this specification. However, it should be noted that the autonomous driving scene perception system 102 in this specification may also include multiple processors 120. Therefore, the operation and / or method steps disclosed in this specification may be performed by one processor as described in this specification, or they may be performed jointly by multiple processors. For example, if the processor 120 of the autonomous driving scene perception system 102 in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 120 (e.g., the first processor executes step A and the second processor executes step B, or the first and second processors jointly execute steps A and B).

[0056] Of course, the modules that implement the autonomous driving scene perception method provided in this specification can also be designed as dedicated circuits, hardware modules and / or devices, independently communicating with the autonomous driving scene perception system 102 or existing as internal components in the autonomous driving scene perception system 102 (not shown in Figure 2).

[0057] The communication module 150 can be connected to the processor 120 for data communication between the autonomous driving scene perception system 102 and external sources, such as servers and databases (not shown in Figure 2). The communication module 150 may include at least one of a wired communication module and a wireless communication module. The internal communication bus 110 can connect different system components, including the storage medium 130, the processor 120, and the communication module 150.

[0058] Figure 2A A flowchart of an autonomous driving scene perception method P100 according to some embodiments of this specification is shown. As previously described, the autonomous driving scene perception system 102 can be used to execute the autonomous driving scene perception method P100 described herein. Specifically, the processor 120 can read an instruction set stored in its local storage medium and then execute the autonomous driving scene perception method P100 described herein according to the provisions of the instruction set.

[0059] like Figure 2A As shown, method P100 may include:

[0060] S110: Acquire target point cloud data of the target scene.

[0061] In some embodiments, the LiDAR can acquire raw 3D point cloud data of an autonomous driving scenario. Figure 3 A schematic diagram of raw point cloud data provided according to some embodiments of this specification is shown. Each point in the raw point cloud data includes three-dimensional coordinates and reflectivity (x, y, z, r) in the lidar coordinate system. In some embodiments, the lidar can send the raw point cloud data to an autonomous driving scene perception system 102, thereby enabling the processor 120 in the autonomous driving scene perception system 102 to acquire the raw point cloud data. The processor 120 can preprocess the raw point cloud data to obtain first projection map data, which is the target point cloud data. In some embodiments, the lidar can preprocess the raw point cloud data to obtain the first projection map data and send the first projection map data to the autonomous driving scene perception system 102. In this way, the processor 120 directly acquires the first projection map data, i.e., directly acquires the target point cloud data, without the processor 120 needing to perform conversion on the raw point cloud data. The first projection map data can be, for example, front view data, top view data, side view data, etc., and can also be projection map data obtained by projecting the raw point cloud data onto other planes. Figure 4 A schematic diagram of front view data provided according to some embodiments of this specification is shown.

[0062] If the processor 120 obtains the original point cloud data, taking the target point cloud data as front view data with size h*w*c as an example, the original point cloud data can be converted into front view data using the following formula:

[0063] Formula 1:

[0064] Where (x, y, z) are the three-dimensional coordinates of the original point cloud data, and r is the reflectance of the point located at (x, y, z) in the point cloud. h and w are the predefined height and width of the front view data, c is the number of channels, and f is the vertical field of view of the LiDAR. down (u,v) represents the angle at which the lidar scans horizontally downwards, and (u,v) represents the coordinates of the front view data.

[0065] After acquiring the target point cloud data, processor 120 continues to execute the following steps of method P100:

[0066] S120: Input the target point cloud data into the target neural network and output the perception results of multiple tasks respectively.

[0067] The processor 120 can output multiple perception results for multiple tasks through a target neural network; therefore, the target neural network can also be called a multi-task neural network. The multiple tasks may include, for example, any number of tasks among semantic segmentation, lane detection, traffic sign detection, object detection, curb detection, and drivable area segmentation. Of course, the multiple tasks may also include other tasks, such as ground elevation regression and road obstacle classification. This specification does not specifically limit the multiple tasks.

[0068] The semantic segmentation includes identifying the category of each point in the target point cloud data, segmenting the regions containing different objects according to the category of each point, and obtaining a semantic segmentation result after region segmentation. The lane line detection includes determining the points in the target point cloud data that are classified as lane lines, detecting multiple lane lines based on the points classified as lane lines, and obtaining a lane line detection result containing multiple lane lines. The traffic sign detection includes determining the points in the target point cloud data that are classified as traffic signs, detecting multiple traffic signs based on the points classified as traffic signs, and obtaining a traffic sign detection result containing multiple traffic signs. The target detection includes detecting multiple target objects from the target point cloud data, determining the category of each target object, and generating a bounding box for each target object, and obtaining a target detection result containing the categories and bounding boxes of multiple target objects. The curb detection includes determining the points in the target point cloud data that are classified as curbs, detecting multiple curb obstacles based on the points classified as curbs, and obtaining a curb detection result containing multiple curb obstacles. The drivable area segmentation includes determining points in the target point cloud data that are classified as drivable areas, segmenting at least one drivable area based on the points classified as drivable areas, and obtaining a drivable segmentation result containing the at least one drivable area.

[0069] The target neural network includes at least one backbone network and multiple branch networks, each of which can handle at least one task. The processor 120 can input target point cloud data into the backbone network, extract shared features from the target point cloud data through the backbone network, and output a shared feature map. Different tasks can share this shared feature map, that is, each branch network can output the perception result of its own processing task based on the shared feature map. In this way, multiple branch networks can output the perception results of multiple tasks.

[0070] Figure 2BA schematic diagram of a multi-level shared backbone network according to some embodiments of this specification is shown. The backbone network may include K levels of cascaded shared networks, where K is an integer greater than or equal to 1, such as 1, 2, 3, or 4. In some embodiments, each level of the shared network uses the output of the previous level's shared network as its input, and then passes its output as input to the next level's shared network. However, the input of some levels of the shared network is not limited to the output of the previous level; it may also have other auxiliary inputs, and its output may not only be output to the next level's shared network, but may also be output to higher or lower level shared networks. Thus, the processor 120 can input the target point cloud data into the K-level shared networks for feature extraction. Each level of the shared network can output a shared feature map for that level. Therefore, the processor 120 can obtain the i-th level shared feature map of the target point cloud data at the output of the i-th level shared network, where i is any value between 1 and K, thereby obtaining the K-level shared feature maps of the target point cloud data, i.e., K shared feature maps. The K-level shared feature map reflects different feature information of the original point cloud data. For example, the K-level shared feature map reflects the feature information of the original point cloud data under different viewpoints, and / or reflects the feature information of the original point cloud data under different depths, etc.

[0071] Accordingly, the multiple branch networks can perform autonomous driving scene perception based on the K-level shared feature map, and output the perception results of multiple tasks respectively. In some embodiments, the processor 120 can divide the multiple branch networks into K classes, each of the K classes of branch networks corresponding to the K-level shared feature map. Each class of branch networks includes one or more branch networks, and each class of branch networks has commonalities, such as being suitable for using raw point cloud data from the same perspective for autonomous driving scene perception. The i-th class of branch networks shares the i-th level shared feature map. Specifically, the processor 120 can input the i-th level shared feature map into all branch networks in the i-th class of branch networks for autonomous driving scene perception, obtain one or more perception results output by the i-th class of branch networks, and then use all the perception results output by the K-class branch networks as the perception results of the multiple tasks.

[0072] Taking K=2 as an example, the backbone network includes two levels of shared networks, namely the first level shared network and the second level shared network, and the multiple branch networks include two types of branch networks, namely the first type of branch network and the second type of branch network. Figure 5 A framework diagram illustrating scene perception for autonomous driving via a Level 2 shared network, according to some embodiments of this specification, is shown. For example... Figure 5As shown, taking the processor 120 acquiring raw point cloud data as an example, the processor 120 can convert the raw point cloud data into target point cloud data (not shown in the figure), and input the target point cloud data into the first-level shared network (using U-net as an example) for feature extraction to obtain the first-level shared feature map. Then, the processor 120 inputs the first-level shared feature map into the first-type branch network for perception, and obtains one or more perception results output by the first-type branch network. For example, the first-type branch network includes three branch networks, corresponding to processing three tasks: semantic segmentation, lane line detection, and traffic sign detection. The first-type branch network may also include one branch network, which processes the three tasks: semantic segmentation, lane line detection, and traffic sign detection. Then, the first-type branch network will output three perception results: semantic segmentation result, lane line detection result, and traffic sign detection result. Figure 6 A schematic diagram illustrating semantic segmentation results provided according to some embodiments of this specification is shown. Figure 7 The diagram illustrates a driving scenario including lane lines according to some embodiments of this specification. Figure 8 This diagram illustrates lane line detection results containing only lane lines, according to some embodiments of this specification. Figure 9 A schematic diagram illustrating traffic sign detection results according to some embodiments of this specification is shown.

[0073] Continue to refer to Figure 5 After acquiring the first-level shared feature map, the processor 120 can further perform a stitching and projection process on the first-level shared feature map and the target point cloud data to obtain second projection map data (taking BEV as an example). This second projection map data is then input into the second-level shared network (FPN as an example) for feature extraction to obtain the second-level shared feature map. Next, the processor 120 inputs the second-level shared feature map into the second-type branch network for perception, obtaining one or more perception results output by the second-type branch network. For example, the second-type branch network also includes three branch networks, each corresponding to one of three tasks: object detection, roadside detection, and drivable area segmentation. Therefore, the second-type branch network will output three perception results: object detection result, roadside detection result, and drivable area segmentation result. Figures 10-12 Schematic diagrams of target detection results, curb detection results, and drivable segmentation results provided according to some embodiments of this specification are shown respectively.

[0074] It should be noted that the stitching and projection processing includes stitching and projection processing. Stitching can be performed first, followed by projection, or vice versa. Specifically, the processor 120 can first perform a concat process on the first-level shared feature map and the target point cloud data to obtain stitched feature map data. Concat refers to stitching the first-level shared feature map and the target point cloud data along the channel dimension, i.e., merging the number of channels, which increases the feature dimension of the feature map while keeping the information content under each dimension unchanged. Then, the processor 120 can perform projection processing on the stitched feature map data to obtain second projection map data. Alternatively, the processor 120 can first perform projection processing on the first-level shared feature map and the target point cloud data separately, and then stitch the projected first-level shared feature map and the projected target point cloud data together to obtain the second projection map data.

[0075] To fully utilize the rich information in the original point cloud data, the second projection map data and the aforementioned first projection map data (target point cloud data) can reflect information about the original point cloud data from different perspectives. The second projection map data can, for example, be front view data, top view data, side view data, etc. Since the first-level shared feature map is obtained by extracting features from the first projection map data through a first-level shared network, the first-level shared feature map and the first projection map data reflect different information about the original point cloud data from the same perspective. For example, if the first projection map data is front view data, the first-level shared feature map is the feature map of the front view data. Correspondingly, since the second-level shared feature map is obtained by extracting features from the second projection map data through a second-level shared network, the second-level shared feature map and the second projection map data also reflect different information about the original point cloud data from the same perspective. For example, if the second projection map data is top view data, the second-level shared feature map is the feature map of the top view data. Therefore, the first-level shared feature map and the second-level shared feature map reflect the feature information of the original point cloud data from different perspectives. For example, the first-level shared feature map includes the feature map of the original point cloud data from the first perspective, and the second-level shared feature map includes the feature map of the original point cloud data from the second perspective.

[0076] The raw point cloud data from different perspectives may include raw point cloud data from a first perspective and raw point cloud data from a second perspective. For example, the raw point cloud data from the first perspective includes: front view data obtained by projecting the raw point cloud data onto a cylindrical surface; the raw point cloud data from the second perspective includes: bird's-eye view data obtained by projecting the raw point cloud data onto a horizontal plane. As another example, the raw point cloud data from the first perspective includes: bird's-eye view data obtained by projecting the raw point cloud data onto a horizontal plane; the raw point cloud data from the second perspective includes: front view data obtained by projecting the raw point cloud data onto a cylindrical surface.

[0077] The advantages of raw point cloud data differ depending on the viewing angle. For example, front-view (range view) data retains more original information from the point cloud data, but suffers from occlusion and scale issues (objects appearing larger when closer to them). Top-view (BEV) data, on the other hand, suffers from information loss but avoids occlusion and scale problems. Furthermore, different tasks have different characteristics. For instance, traffic signs face the direction of LiDAR scanning, meaning front-view data is sufficient for traffic sign detection, while top-view data fails to identify them. Similarly, pedestrians around an autonomous vehicle may be occluded by other objects. Front-view data may fail to identify occluded pedestrians, posing a safety risk, while top-view data can identify them. Therefore, top-view data is more suitable for object detection tasks. Thus, the processor 120 can deploy appropriate branch networks at the output of different levels of the shared network, taking into account the characteristics of different tasks and the advantages of raw point cloud data from different viewing angles. In other words, the deployment of branch networks is related to the feature map of the raw point cloud data from a specific viewpoint, enabling each branch network to use appropriate feature maps for autonomous driving scene perception, thereby improving the accuracy of autonomous driving scene perception. Figure 5 In this study, the first type of branch network is more suitable for using feature maps of the original point cloud data from the first perspective, so both are based on the original point cloud data from the first perspective for perception. The second type of branch network is more suitable for using feature maps of the original point cloud data from the second perspective, so both are based on the original point cloud data from the second perspective for perception. Figure 13 A framework diagram of semantic segmentation and object detection based on target point cloud data, according to some embodiments of this specification, is shown. Figure 13 As shown, the target point cloud data is front view data. The first-level shared feature map is obtained by extracting features from the front view data using the first-level shared network U-net. Taking a branch network of type 1 that only includes semantic segmentation as an example, the semantic segmentation task is directly performed based on the first-level shared feature map. Taking a branch network of type 2 that only includes object detection as an example, the first-level shared feature map is processed by stitching and projection to obtain the second-level shared feature map (BEV), and then the object detection task is performed based on the second-level shared feature map.

[0078] The first-level sharing network mentioned above can be an FCN (Fully Convolutional Neural Network), U-net, U-net++, or similar networks. Taking U-net as an example... Figure 14 A U-net structure diagram is shown according to some embodiments provided in this specification. For example... Figure 14As shown, U-net mainly consists of three parts: an encoder network, a decoder network, and skip connections. The left half is the encoder network, which reduces the size of the feature map and extracts some shallow features through convolution and downsampling. To reduce network parameters and improve the network's generalization ability, the left half can use pooling layers to reduce the dimensionality of the feature map, i.e., downsampling. The right half is the decoder network, which obtains some deep features through convolution and upsampling, so that the output of the last layer of the decoder network is a first-level shared feature map with the same size as the input target point cloud data. In the middle, the feature maps obtained in the encoding stage are combined with those obtained in the decoding stage through connections. By combining deep and shallow features, the decoder network can share the feature maps learned by the encoder network segment, significantly reducing network parameters and facilitating real-time application deployment.

[0079] The result of inputting the target point cloud data I into U-net can be represented by the following formula:

[0080] Formula 2: y'=f(I;(W,b))=W N H N-1

[0081] Where y' is the output of the last layer of the encoding network, W and b are the weights and biases in the U-net network, f(·) is the activation function, which can be the ReLU (Rectified Linear Unit) function, H is the output of the hidden layer, and H0 is the target point cloud data I. For all weights and biases, the output H of the nth hidden layer (n∈{1,2,...,N-1}) is... n It can be calculated using the following formula three:

[0082] Formula 3: H n =pool(f(W) n H n-1 +b n ))

[0083] Here, pool(·) is the pooling function.

[0084] After passing through the encoding network, upsampling is required through the decoding network, which can be calculated using the following formula:

[0085] Formula 4: y = upsample(y')

[0086] Here, upsample(·) is the upsampling function, and y is the output of the last layer of the decoding network, i.e., the first-level shared feature map. The number k of the first-level shared feature maps will vary depending on the design of the U-net parameters.

[0087] The aforementioned Level 2 sharing network can employ networks such as FPN (Feature Pyramid Network) and RPN (Region Proposal Network). Taking FPN as an example... Figure 15 A schematic diagram of an FPN structure according to some embodiments of this specification is shown. Figure 15 As shown, the FPN includes a bottom-up path, a top-down path, and lateral connections. The bottom-up process is the forward pass of the FPN network. During this forward pass, the size of the second projection map data changes after passing through some network layers, but remains unchanged after passing through others. Layers that do not change the feature map size can be grouped into a single stage. The extracted features are the output of the last network layer in each stage, thus forming a feature pyramid. The top-down process uses upsampling. The lateral connection merges the upsampled feature map with a feature map of the same size generated from the bottom-up process, thus fusing high-resolution shallow features with deep features rich in semantic information to output a second-level shared feature map.

[0088] It should be noted that the processor 120 can linearly weight the multiple loss functions of the multiple branch networks to obtain a joint loss function, and then use the joint loss function to iteratively train the target neural network. When the joint loss function reaches its optimal solution, the final trained target neural network is obtained. The optimal solution of the joint loss function is the best solution obtained by integrating the collective model features of all branch networks, and is the best solution after integration. Conversely, the optimal solution of the loss function for a single branch network is only the optimal solution for that single branch network, and the network parameters under this optimal solution may not be suitable for other branch networks.

[0089] Because the joint loss function integrates information from all tasks within the entire target neural network architecture, parameters across multiple tasks become correlated. Each branch network can incorporate auxiliary information from other branches during training, leading to better training of each branch. For example, if the first type of branch network includes a semantic segmentation network and the second type includes an object detection network, the joint loss function combines the loss functions of both networks, creating a correlation and mutual reinforcement between their parameters. Since semantic segmentation requires determining the category of each point, while object detection can detect multiple objects by referencing the category of each point, it is evident that utilizing features extracted from semantic segmentation can improve detection performance.

[0090] The processor 120 can obtain the joint loss function by linearly weighting the multiple loss functions of the multiple branch networks using the following formula:

[0091] Formula 5:

[0092] Where L is the joint loss function, M is the number of the multiple tasks, and L m Let λ be the loss function for the m-th task. m The weight for the m-th task can be assigned to each branch network based on the importance of different tasks. The semantic segmentation network can use a cross-entropy loss function and a fractional regression loss function. The lane detection network, traffic sign detection network, and curb detection network can all use the same loss function, such as the center loss function. The object detection network can use a focal loss function and a SmoothL1 loss function. The drivable area segmentation network can also use a cross-entropy loss function, as shown in Formula 6.

[0093] Formula 6: L1 = -y*log(p) - (1-y)*log(1-p)

[0094] Where L1 is the cross-entropy loss function, y is the truth label, y is 1 for drivable areas and 0 for indestructible areas, and p is the prediction probability.

[0095] The ground height regression can employ the L2 loss function, i.e., the Mean Square Error (MSE) function, as expressed in Formula 7 below:

[0096] Formula 7:

[0097] Among them, y i f(x) represents the actual ground height. i ) is the predicted height value.

[0098] As mentioned above, the processor 120 can first train the target neural network and then use the finally trained target neural network to implement the autonomous driving scene perception method of this specification. Figure 16 A flowchart of a method P200 for training a target neural network in an autonomous driving scenario, according to some embodiments of this specification, is shown. As previously described, the autonomous driving scenario perception system 102 can be used to train the target neural network. Specifically, the processor 120 can read an instruction set stored in its local storage medium and then execute the method P200 for training a target neural network in an autonomous driving scenario as described in this specification, according to the instructions in the instruction set. Figure 16 As shown, method P200 may include:

[0099] S210: Acquire point cloud data collected by the lidar mounted on the vehicle.

[0100] S220: Construct a multi-task neural network for autonomous driving scenarios.

[0101] S230: Set the learning strategy and loss function for the training phase, train the neural network constructed in S220 using the training set, and test the training effect of the neural network using the test set.

[0102] S240: Save the target neural network after training and testing, verifying that its performance meets the preset indicators.

[0103] S250: Based on the target neural network trained by S240, it applies the target point cloud data to be perceived to obtain multi-task perception results in autonomous driving scenarios.

[0104] In summary, the autonomous driving scene perception method P100 and system 102 of this specification provide a point cloud-based target neural network that couples multiple tasks into the same neural network framework for processing. Different tasks share the same backbone network and the same feature information, allowing multiple perception tasks to be implemented simultaneously. This improves the network's real-time performance and allows for the fulfillment of various scene perception requirements for autonomous driving within a single framework. Moreover, coupling multiple tasks into the same target neural network framework not only reduces computational load but also fully utilizes information from different tasks for training and iteration. Different tasks and functions are interconnected and mutually reinforcing.

[0105] The autonomous driving scene perception method P100 and system 102 described in this specification employ feature maps from different perspectives to suit the characteristics of different tasks. For example, tasks such as semantic segmentation, lane detection, and traffic sign detection are processed based on the first-level shared feature map in the first perspective, while tasks such as target detection, roadside detection, and drivable area segmentation are processed based on the second-level shared feature map in the second perspective. By making full use of the advantages of feature maps from different perspectives, a multi-view, multi-task autonomous driving scene perception method is constructed.

[0106] The autonomous driving scene perception method P100 and system 102 in this specification iteratively train the entire autonomous driving scene perception framework through a joint loss function composed of multiple branch networks. While realizing multiple functions, the correlation of parameters between multiple tasks helps to better train the branch networks of each task.

[0107] The autonomous driving scene perception method P100 and system 102 described in this specification utilize point cloud data, which contains richer 3D information than images. The autonomous driving scene perception method based on point cloud is more reliable than image algorithms and can achieve multiple perception requirements for autonomous driving within the same target neural network framework.

[0108] This specification, in another aspect, provides a non-temporary computer-readable medium storing at least one set of instructions for autonomous driving scene perception. When the at least one set of instructions is executed by a processor, it instructs the processor to perform the steps of the autonomous driving scene perception method P100 described herein. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on an autonomous driving scene perception system 102, the program code is used to cause the autonomous driving scene perception system 102 to perform the steps of the autonomous driving scene perception method P100 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the autonomous driving scene perception system 102. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing the operations described herein may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages.The program code can be executed entirely on the autonomous driving scene perception system 102, partially on the autonomous driving scene perception system 102, as a standalone software package, partially on the autonomous driving scene perception system 102 and partially on a remote computing device, or entirely on a remote computing device.

[0109] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0110] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0111] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0112] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and aiding in the understanding of a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art may readily identify some of the devices as separate embodiments when reading this specification. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. It is also valid when each secondary embodiment contains fewer than all the features of a single foregoing disclosed embodiment.

[0113] Each patent, patent application, publication of the patent application, and other materials such as articles, books, specifications, publications, documents, articles, etc., cited herein may be incorporated by reference. The entire contents used for all purposes, except for any history of prosecution documents associated with it, that may be inconsistent with or conflict with this document, or that may have a limiting effect on the widest extent of the claims, are now or hereafter associated with this document. For example, in the event of any inconsistency or conflict between the description, definition, and / or use of terms associated with any of the included materials and the terms, description, definition, and / or used in connection with this document, the terms used herein shall prevail.

[0114] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A method for scene perception in autonomous driving, characterized in that, include: Acquire target point cloud data of the target scene; as well as The target point cloud data is input into a target neural network, which outputs the perception results of multiple tasks, wherein the target neural network includes at least: A backbone network is configured to extract shared features from the target point cloud data and output a shared feature map. The backbone network includes multiple cascaded shared networks, comprising a first-level shared network and a second-level shared network. The first-level shared network is configured to extract features from the target point cloud data to obtain a first-level shared feature map. The second-level shared network is configured to extract features from second projection map data to obtain a second-level shared feature map. The second projection map data is obtained by stitching and projecting the first-level shared feature map onto the target point cloud data. Multiple branch networks, each configured to output the perception results of the multiple tasks based on the shared feature map.

2. The method as described in claim 1, characterized in that, The acquisition of target point cloud data for the target scene includes: The three-dimensional raw point cloud data obtained by lidar scanning is acquired, the three-dimensional raw point cloud data is preprocessed to obtain the first projection map data, and the first projection map data is determined as the target point cloud data.

3. The method as described in claim 2, characterized in that, The multi-level cascaded shared network is a K-level cascaded shared network, where K is an integer greater than or equal to 2.

4. The method as described in claim 3, characterized in that, The step of inputting the target point cloud data into the target neural network includes: The target point cloud data is input into a K-level shared network for feature extraction. The i-th level shared feature map of the target point cloud data is obtained at the output of the i-th level shared network, where 1≤i≤K.

5. The method as described in claim 4, characterized in that, The first-level shared feature map includes the feature map of the original point cloud data from the first viewpoint; and The second-level shared feature map includes the feature map of the original point cloud data from the second perspective.

6. The method as described in claim 5, characterized in that, The original point cloud data in the first perspective includes: front view data obtained by projecting the original point cloud data onto a cylindrical surface.

7. The method as described in claim 5, characterized in that, The plurality of branch networks includes a first type of branch network, which includes at least one branch network, and the output of the perception results of the plurality of tasks includes: The first-level shared feature map is input into the first-type branch network for perception, and at least one perception result output by the first-type branch network is obtained. The first-type branch network performs perception based on the original point cloud data from the first viewpoint. The first type of branch network is configured to process at least one of the following tasks: semantic segmentation, lane line detection, and traffic sign detection, wherein the at least one perception result includes at least one of semantic segmentation result, lane line detection result, or traffic sign detection result.

8. The method as described in claim 5, characterized in that, The original point cloud data in the second perspective includes: bird's-eye view data obtained by projecting the original point cloud data onto a horizontal plane.

9. The method as described in claim 5, characterized in that, The multiple branch networks include a second type of branch network, which includes at least one branch network, and the output of the perception results for each of the multiple tasks includes: The second-level shared feature map is input into the second-type branch network for perception, and at least one perception result output by the second-type branch network is obtained. The second-type branch network performs perception based on the original point cloud data from the second viewpoint. The second type of branch network processes at least one of the following tasks: target detection, roadside detection, and drivable area segmentation, wherein the at least one perception result includes at least one of the following: target detection result, roadside detection result, or drivable area segmentation result.

10. The method as described in claim 1, characterized in that, The target neural network is a network obtained by iterative training using a joint loss function, which is obtained by linearly weighting multiple loss functions of the multiple branch networks.

11. An autonomous driving scene perception system, comprising: At least one storage medium stores at least one set of instructions for autonomous driving scenario perception; as well as At least one processor is communicatively connected to the at least one storage medium. When the autonomous driving scene perception system is running, the at least one processor reads the at least one instruction set and implements the autonomous driving scene perception method according to any one of claims 1 to 10.

12. A non-temporary computer-readable medium, characterized in that, It stores at least one set of instructions for autonomous driving scene perception, wherein when the at least one set of instructions is executed by the processor, it instructs the processor to perform the autonomous driving scene perception method as described in any one of claims 1 to 10.

Images