Perception method and device based on point cloud scene, mobile tool and storage medium

Abstract

The invention discloses a perception method and device based on a point cloud scene, a mobile tool and a storage medium, and the method comprises the steps: obtaining time series data of a current point cloud scene, the time series data comprising n frames of point cloud data continuously collected up to a current moment, and n being a positive integer greater than 1; an aerial view grid expression of the time sequence data is obtained, and the aerial view grid expression is represented by aerial view grid matrixes in one-to-one correspondence with all frames of point cloud data in the time sequence data; and obtaining a sensing result corresponding to the time sequence data of the current point cloud scene based on the aerial view grid expression and a pre-trained network model. According to the method, the obtained n frames of point cloud data which are continuously collected are used as the input of the model, so that space information and time information can be fully considered when the sensing result is predicted, the accuracy and reliability of the sensing result are improved, and subsequent operation of track prediction, path planning and the like is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a point cloud scene-based perception method, device, mobile tool and storage medium. Background Art

[0002] With the development of LiDAR and artificial intelligence technology, driverless technology has made great progress and has been widely used in driverless taxis, high-level assisted driving and other fields. On the one hand, driverless technology should liberate the driver's hands and make driving simpler and more convenient; on the other hand, it should also ensure the safety of driving, not only the safety of its own vehicle and passengers, but also the safety of vehicles and pedestrians in the environment. Therefore, it is required that the autonomous driving vehicle can have a full and accurate perception of the environment in which it is located. In the existing perception algorithm, its input is mainly a single-frame image. Since the single-frame image is static data information, the corresponding perception algorithm can only obtain static perception results, resulting in insufficient perception of the environment in which it is located, and it is easy to affect the reliability and accuracy of the perception results due to defects in the single-frame image. At the same time, the existing perception algorithms are mainly based on target detection, semantic segmentation and target tracking. However, most of these perception algorithms realize a single function through a perception algorithm. Therefore, if target detection, semantic segmentation and target tracking are to be realized at the same time, it is necessary to develop multiple independent perception algorithms, which will consume and occupy a lot of computing resources. Summary of the invention

[0003] An embodiment of the present invention provides a point cloud scene-based perception method to at least solve the problem in the prior art that the perception algorithm completely relies on a single-frame image, resulting in incomplete and inaccurate perception of the environment.

[0004] In a first aspect, an embodiment of the present invention provides a point cloud scene-based perception method, comprising:

[0005] Acquire time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data collected continuously up to the current moment, wherein n is a positive integer greater than 1;

[0006] Acquire a bird's-eye view grid expression of the time series data, wherein the bird's-eye view grid expression is represented by a bird's-eye view grid matrix corresponding one-to-one to each frame of point cloud data in the time series data;

[0007] Based on the bird's-eye view grid expression and the pre-trained network model, the perception result corresponding to the time series data of the current point cloud scene is obtained.

[0008] In a second aspect, an embodiment of the present invention provides a perception device based on a point cloud scene, comprising:

[0009] A time series data acquisition module, used to acquire time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data continuously collected up to the current moment, wherein n is a positive integer greater than 1;

[0010] A bird's-eye view rasterization module, used to obtain a bird's-eye view raster expression of the time series data, wherein the bird's-eye view raster expression is represented by a bird's-eye view raster matrix corresponding one-to-one to each frame of point cloud data in the time series data;

[0011] The perception prediction module is used to obtain the perception result based on the bird's-eye view grid expression and the pre-trained network model unit.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0013] A memory for storing executable instructions; and

[0014] A processor is used to execute executable instructions stored in a memory, wherein the executable instructions, when executed by the processor, implement the steps of the point cloud scene-based perception method of the first aspect described above.

[0015] In a fourth aspect, an embodiment of the present invention provides a mobile tool, which includes: the above-mentioned point cloud scene-based perception device or the above-mentioned electronic device in the third aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a mobile tool, the mobile tool comprising:

[0017] A collection device for collecting real-time point cloud data from the surrounding environment;

[0018] A controller, used to obtain n frames of point cloud data continuously collected up to the current moment from the acquisition device as time series data of the current point cloud scene, and obtain a bird's-eye view grid expression of the time series data, and obtain a perception result of the current point cloud scene at the current moment based on the bird's-eye view grid expression of the time series data and a pre-trained network model, wherein n is a positive integer greater than 1, and the bird's-eye view grid expression is represented by a bird's-eye view grid matrix corresponding to each frame of point cloud data in the time series data;

[0019] A planner is used to plan and control the mobile tool according to the perception results output by the controller.

[0020] In a sixth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the point cloud scene-based perception method of the first aspect described above.

[0021] In the seventh aspect, an embodiment of the present invention provides a computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the point cloud scene-based perception method of the first aspect mentioned above.

[0022] The beneficial effect of the embodiment of the present invention is that the method provided by the embodiment of the present invention uses the continuously collected n-frame point cloud data as the basic information for obtaining the perception result at the current moment, so that the spatial information and time information can be fully considered when predicting the perception result. Compared with the perception algorithm of the prior art that performs perception prediction based on a single frame image as input, the time series data including the continuous n-frame point cloud data used by the present invention can represent more data information (that is, the feature expression is richer and includes both spatial and temporal information), so the perception prediction based on it can obtain more accurate perception results, effectively improve the calculation accuracy of the perception algorithm, and make it possible to obtain more perception results of different forms, which is beneficial to the subsequent trajectory prediction, path planning and other calculations, and improves the safety of autonomous driving. In addition, the embodiment of the present invention also performs perception calculations by obtaining the bird's-eye view grid expression of the time series data, so as to two-dimensionalize the data while maintaining the key variables in the data, thereby effectively reducing the calculation amount of the perception algorithm and improving the calculation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0024] Figure 1 This is a flow chart of a point cloud scene perception method according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of step S12 of the point cloud scene perception method according to an embodiment of the present invention;

[0026] Figure 3 This is a flowchart of step S13 of the point cloud scene perception method according to an embodiment of the present invention;

[0027] Figure 4 This is a principle block diagram of a point cloud scene-based perception device according to an embodiment of the present invention;

[0028] Figure 5This is a principle block diagram of a point cloud scene-based perception device according to another embodiment of the present invention;

[0029] Figure 6 This is a principle block diagram of a mobile tool according to an embodiment of the present invention;

[0030] Figure 7 is a principle block diagram of a mobile tool according to another embodiment of the present invention;

[0031] Figure 8 It is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0034] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0035] In the present invention, "module", "device", "system" and the like refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, an element can be, but is not limited to, a process, a processor, an object, an executable element, an execution thread, a program and / or a computer running on a processor. In addition, an application or script program running on a server, a server can all be an element. One or more elements can be in an execution process and / or thread, and an element can be localized on a computer and / or distributed between two or more computers, and can be operated by various computer-readable media. An element can also communicate through local and / or remote processes according to a signal with one or more data packets, for example, a signal from a data that interacts with another element in a local system, a distributed system, and / or a network on the Internet through a signal to interact with other systems.

[0036] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0037] The point cloud scene-based perception method in the embodiment of the present invention can be applied to a point cloud scene-based perception device, so that users can use the point cloud scene-based perception method to obtain more accurate perception results that consider more feature expressions, and can also obtain more perception results of different dimensions based on the method, thereby improving the computing efficiency and accuracy of perception. These point cloud scene-based perception devices include but are not limited to perception modules on autonomous driving equipment, controllers on autonomous driving equipment, smart phones, smart tablets, personal PCs, computers, cloud servers, etc., and the present invention does not limit this. In particular, the point cloud scene-based perception method in the embodiment of the present invention can also be applied to autonomous driving vehicles (such as passenger cars, buses, minibuses, buses, trucks, sanitation vehicles, sweepers, floor scrubbers, etc.), sweeping robots, motorcycles, electric vehicles and other autonomous driving equipment, and the present invention does not limit this.

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0039] Figure 1 The point cloud scene perception method according to an embodiment of the present invention is schematically shown. Figure 1 As shown, it includes the following steps:

[0040] Step S11: acquiring time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data continuously collected up to the current moment, wherein n is a positive integer greater than 1;

[0041] Step S12: obtaining a bird's-eye view grid expression of the time series data, wherein the bird's-eye view grid expression is represented by a bird's-eye view grid matrix corresponding one-to-one to each frame of point cloud data in the time series data;

[0042] Step S13: Based on the bird's-eye view grid expression and the pre-trained network model, the perception result of the current point cloud scene at the current moment is obtained.

[0043] In step S11, the time series data at least includes the point cloud data of the current point cloud scene continuously collected up to the current moment, and may also be the point cloud data continuously collected within a required period of time that can be used to predict and perceive the current point cloud scene. Since it is point cloud data collected continuously over a period of time, rather than single-frame point cloud data, such time series data can simultaneously express the time and space information of the current point cloud scene. Among them, the point cloud scene refers to a scene for acquiring and expressing surrounding environment information based on point cloud data, and the current point cloud scene refers to the point cloud scene corresponding to the current moment. As a possible implementation method, the time series data may specifically include n frames of point cloud data collected continuously up to the current moment, which may be collected by a laser point cloud radar or other equipment provided on the autonomous driving vehicle. It is understandable that each frame of point cloud data contains a number of point cloud data, n frames represent the total number of frames collected continuously, and n is a positive integer greater than 1. Taking the time series data of the current point cloud scene obtained in the embodiment of the present invention as represented by P as an example, P t-n:t It represents the n-frame continuous point cloud of the current point cloud scene collected from time tn to time t, where tn represents the starting time of collecting the n-frame continuous point cloud, and t represents the ending time of collecting the n-frame continuous point cloud, which preferably refers to the current time.

[0044] In order to enable the perception algorithm to effectively identify the environmental feature information expressed in the point cloud data, in step S12, the time series data obtained in step S11 needs to be preprocessed to convert the three-dimensional point cloud data into two-dimensional projection data on the bird's-eye view. As a possible implementation, the embodiment of the present invention converts the three-dimensional point cloud data into two-dimensional projection data on the bird's-eye view by rasterization coding. Preferably, the two-dimensional projection data of the time series data obtained by rasterization coding is a bird's-eye grid expression of the time series data. Figure 2The process of step S12 in the point cloud scene perception method according to an embodiment of the present invention is schematically shown. Figure 2 As shown, the method can be specifically implemented by comprising the following steps:

[0045] S21: Projecting each frame of point cloud data in the time series data onto the XY plane through rasterization encoding;

[0046] S22: Obtaining a bird's-eye view grid matrix corresponding to the corresponding frame point cloud data according to the projection results of each frame point cloud data.

[0047] In step S21, the XY plane is first rasterized according to preset rules to form a rasterized bird's-eye view (also referred to as a bird's-eye view in the embodiment of the present invention). The preset rules can be set according to the actual application scenario, and are mainly rules for rasterization, such as the size of the grid obtained after rasterization. Then, the point clouds in each frame of point cloud data are projected onto the rasterized XY plane (i.e., the rasterized bird's-eye view) according to the coordinates of each point cloud in each frame of point cloud data in the time series data, so as to obtain the projection of each frame of point cloud data on a plane perpendicular to the height direction. It can be understood that after processing in step S21, each frame of point cloud data in the time series data corresponds to a projection on the XY plane, that is, there are also n projections corresponding to n frames of point cloud data on the XY plane.

[0048] After obtaining the projection of each frame of point cloud data on the XY plane, in step S22, a bird's-eye view grid expression of the time series data will be obtained according to the projection results of each frame of point cloud data. The bird's-eye view grid expression can be specifically represented by a bird's-eye view grid matrix corresponding one-to-one to each frame of point cloud data in the time series data, that is, the bird's-eye view grid expression corresponding to the time series data includes n bird's-eye view grid matrices corresponding one-to-one to n frames of point cloud data. As a possible implementation, the bird's-eye view grid matrix in the embodiment of the present invention can be a w×h×k matrix, wherein w represents the width of a single grid in the rasterized bird's-eye view formed after rasterization encoding, h represents the height of a single grid in the rasterized bird's-eye view, and k represents the characteristic dimension of the point cloud falling into the corresponding grid. Therefore, in step S22, the width w and height h of each grid in the rasterized bird's-eye view can be obtained according to the rasterization process of step S21. At the same time, the rasterized bird's-eye view, that is, the point cloud falling in each grid on the rasterized XY plane can also be obtained according to the projection result of step S21. Specifically, after projecting each point cloud onto the XY plane based on the coordinates of the point cloud itself, since the XY plane is rasterized, it is not difficult to understand that each point cloud projected on the XY plane falls into the corresponding grid. Therefore, the association relationship between each point cloud and the grid in each frame of point cloud data after projection can be determined according to the coordinates of the point cloud itself and the position of the grid in the XY plane, and the point cloud falling in each grid on the XY plane can be determined according to the association relationship. Afterwards, in step S22, the point cloud feature dimension k of each grid on the XY plane can be calculated respectively, thereby forming a bird's-eye view grid matrix w*h*k corresponding to each frame of point cloud data, and the bird's-eye view grid matrix formed thereby expresses the rasterization information (i.e., the width w and height h of the grid) and point cloud feature information (i.e., the feature dimension k of the point cloud corresponding to each grid under the projection of the point cloud data of the corresponding frame) of each frame of point cloud data on the bird's-eye view. As a possible implementation, calculating the point cloud feature dimension k of each grid on the XY plane can be achieved by inputting the height of the point cloud falling into the corresponding grid into a preset encoding function, wherein the preset encoding function can take the height of each point cloud in the corresponding grid as input and the sum of the heights of all point clouds falling into the corresponding grid as output, thereby, by determining the point cloud falling into each grid and inputting the height of each point cloud in each grid into the preset encoding function, the feature dimension k of the point cloud corresponding to each grid can be obtained. In this design, the characteristic dimension k of each grid can reflect the height information of each grid, so that it can be combined with the width information and height information of the grid, and the information of each grid in the formed bird's-eye view grid matrix can be expressed by w*h*k, which is more conducive to the recognition of the algorithm. Preferably, the preset encoding function can be specifically implemented as the following formula:

[0049] f=∑z

[0050] Among them, z represents the height of each point cloud in the grid, and f is the characteristic dimension of the point cloud in the grid, which is represented by the sum of the heights of the point clouds falling into the corresponding grid.

[0051] Therefore, the bird's-eye view grid matrix obtained in the embodiment of the present invention is represented by B as an example, B t-n:t It represents the n bird's-eye view grid matrices corresponding one to one to the n frames of continuous point clouds of the current point cloud scene collected from time tn to time t, that is, the obtained bird's-eye view grid can be expressed as B t-n:t To express.

[0052] It should be noted that the specific execution method of step S21 and step S22 is not limited by the embodiment of the present invention. In some embodiments, it can be implemented by first projecting all n frames of point cloud data in the time series data through step S21, and then determining the bird's-eye view grid matrix corresponding to each frame of point cloud data through the processing of step S22, thereby forming a bird's-eye view grid expression of the time series data; and in other embodiments, it can also be implemented by first projecting one frame of point cloud data in the time series data through step S21 each time, and then determining the bird's-eye view grid matrix corresponding to the currently projected frame of point cloud data through the processing of step S22, and looping the processing of step S21 and step S22 until all n frames of point cloud data in the time series data have been projected and n bird's-eye view grid matrices corresponding to the n frames of point cloud data have been determined.

[0053] In step S13, the pre-trained network model is preferably a network model for multi-task perception, which can be a multi-task perception model with a bird's-eye view grid expression of time series data as input and at least two perception results as output. With this network model, data can be input only once to output a variety of different perception results, so as to save computing resources and improve perception efficiency. As another possible implementation, in step S13, the pre-trained network model can also be a network model composed of a trunk network and at least one branch network. Among them, the bird's-eye view grid expression is used as the input of the trunk network, the output result of the trunk network is used as the input of the branch network, and the output of the branch network is the desired perception result of the surrounding environment, thereby extracting image features from the input bird's-eye view grid expression through the same trunk network, and regressing the extracted features using the branch network, thereby realizing branch prediction of the perception result in an independent sub-task manner, so that the input information of the perception algorithm can be enriched, and the output result of the perception algorithm can be flexibly designed, thereby improving the computing efficiency, perception efficiency and accuracy of the prediction of the perception result of the trained network model.

[0054] As a preferred embodiment, each branch network in the network model can be designed to output a different type of branch prediction result, thereby realizing the design of the corresponding number and type of branch networks according to the needs and expectations to obtain the desired perception results. For example, multiple branch networks can be designed to achieve multi-task output, or only one branch network can be designed to achieve a specific perception prediction. Thus, after obtaining the bird's-eye view grid expression of the time series data, the data content of the input part of the pre-trained network model is obtained. Therefore, in step S13, by inputting the bird's-eye view grid expression into the pre-trained network model, the perception result of the current point cloud scene at the current moment can be obtained based on the output result of the network model. Taking the designed network model as an example, which includes a backbone network and multiple branch networks, in step S13, by inputting the bird's-eye view grid expression into the backbone network, multiple different perception results of the current point cloud scene at the current moment can be obtained based on the output results of each branch network. Preferably, the perception result may include a static perception result and a dynamic perception result. The static perception result may include at least one of a target detection result and a foreground segmentation result. The target detection result is used to determine the position and length, width and height attributes of the detection target in the surrounding environment. Specifically, the rectangular box (bounding box) of the detection target can be used to determine the position and length, width and height attributes of the detection target in the surrounding environment. box), the foreground segmentation result is used to segment the environmental information and detection target in the foreground, which can be specifically represented by the segmentation mask of the foreground; the dynamic perception result may include at least one of the dynamic and static detection results, the motion prediction results, and the relative displacement of the vehicle between the point cloud data of every two consecutive frames. The dynamic and static detection results are used to describe the motion state of the detection target, and the motion state can be specifically represented by dynamic or static. The perception of the dynamic and static detection results can provide more basis for downstream processing to ensure the reliability and safety of downstream planning or control; the motion prediction result is used to describe the movement speed and direction of the object in the scene, which can be specifically represented by a specific speed value and direction angle. The perception of the motion prediction result can enable the downstream algorithm to avoid obstacles earlier to ensure the safety and comfort of downstream planning or control; by perceiving the relative displacement between the continuously collected n frames of point cloud data, it can be convenient for the downstream module to estimate the motion state and driving trajectory of the vehicle according to the relative displacement.

[0055] Therefore, the embodiment of the present invention can train a network model with the bird's-eye view grid expression of time series data as input and the static perception result and dynamic perception result as output, so as to realize the prediction of the motion state and other motion information of the perceived target based on the continuous multi-frame point cloud data of the current scene collected, and make the perception result predicted based on the multi-frame point cloud data, so as to avoid the disadvantages of easy error in perception prediction based on single-frame point cloud data and low reliability of perception result, thereby improving the reliability of perception result, facilitating the subsequent trajectory prediction, path planning and other operations, improving the safety of automatic driving, and improving the accuracy of operation. As a preferred embodiment, in order to better improve the efficiency of perception operation, when training the network model, it can also be trained to enable it to output as many perception results as possible, such as target detection results, foreground segmentation results, dynamic and static detection results, motion prediction results, and vehicle relative displacement between every two consecutive frames of point cloud data as the output of the network model. By increasing the number of perception results of different forms obtained, the accuracy and reliability of trajectory prediction, path planning and other operations can be further improved, the safety of automatic driving can be improved, and the efficiency of perception operation can also be improved.

[0056] Preferably, when the network model is pre-trained, the selected network model can be a network model with a spatiotemporal convolutional neural network as the backbone network. By pre-training the spatiotemporal convolutional neural network model, it becomes a backbone network for feature fusion of spatial information and temporal information based on time series data. As a possible implementation, the trained backbone network is a spatiotemporal convolutional neural network with the bird's-eye view grid expression of the time series data obtained in step S12 as input and the feature map as output. Among them, the training method of the backbone network can be implemented by adopting any existing training method of the spatiotemporal convolutional neural network model, and the embodiment of the present invention does not limit this. The spatiotemporal convolutional neural network is a classic method for processing video data. By training it, the spatiotemporal convolutional neural network can obtain a feature map that can fuse spatial information and temporal information based on time series data. It can be understood that the feature map is high-dimensional data that fuses spatial information and temporal information. By performing perceptual prediction through the feature map, perceptual results of more dimensions can be obtained.

[0057] Furthermore, when the network model is pre-trained, the selected network model can also be a network model with a deep convolutional neural network as a branch network. The deep convolutional neural network is pre-trained to make it a branch network for perceiving and predicting the surrounding environment based on features that integrate spatial information and temporal information. As a possible implementation method, the trained branch network is a deep convolutional neural network that takes the feature map that integrates spatial information and temporal information output by the trunk network as input and the branch prediction result as output. Specifically, the deep convolutional neural network can be trained according to the needs and goals to obtain a branch network that meets the expectations. It can be understood that the branch network that meets the expectations means that the type of perception result corresponding to its branch prediction result is in line with expectations. Taking the type of perception result corresponding to the branch prediction result output by the expected branch network as the target detection result as an example, the deep convolutional neural network can be trained accordingly to make it a branch network that takes the feature map that integrates spatial information and temporal information output by the trunk network as input and the branch prediction result of the target detection result as output. Therefore, according to the type of perception results desired, the branch networks obtained by training the deep convolutional neural network may include a target detection branch network for obtaining target detection results in static perception results, a foreground segmentation branch network for obtaining foreground segmentation results in static perception results, a dynamic and static detection branch network for obtaining dynamic and static detection results in dynamic perception results, a motion prediction branch network for obtaining motion prediction results in dynamic perception results, and an inter-frame displacement prediction branch network for obtaining the relative displacement of the vehicle between point cloud data of each two consecutive frames in the dynamic perception results.

[0058] Among them, in a specific implementation, the structure of the selected network model can be designed and trained accordingly according to the expectations of the functional characteristics of the pre-trained network model, such as the expectations of the perception results that the trained network model can output. For example, the structure of the selected network model can be designed to include a trunk network and a branch network, or a trunk network and two branch networks, or a trunk network and five branch networks, and the embodiment of the present invention does not limit this. It should be particularly noted that no matter what kind of network model is selected, after training it, as long as it can take the bird's-eye view grid expression corresponding to the time series data as input and output the desired perception results, it should be regarded as within the scope of protection of the present invention. As a possible implementation method, taking the selected network model as an example that includes a trunk network and at least one branch network, when training the selected network model, the trunk network and each branch network can be trained separately, or the trunk network and each branch network can be trained at the same time, or the trunk network can be trained separately and each branch network can be trained at the same time. Preferably, when training the selected network model, the backbone network and various branch networks of the network model are trained simultaneously. In this case, the specific training method can be implemented with reference to the prior art, and the embodiment of the present invention will not be described in detail. The difference from the prior art is that when training the selected network model, the embodiment of the present invention will design corresponding loss functions for the backbone network and various branch networks respectively, and also set a weight for each branch network respectively. When performing regression, the embodiment of the present invention not only calculates the loss of the backbone network and various branch networks respectively according to the loss functions designed for them, but also calculates the loss of the entire trained network model based on the loss and weight of each branch network, and only when the losses of the backbone network, various branch networks and the entire network model meet the target, it is determined that the training of the network model is completed. Among them, the loss of each branch network can be determined by the loss function pre-designed for each branch network, and the weight of each branch network can be pre-set in combination with the characteristics of the branch network according to the needs and expectations. The loss of the entire trained network model is calculated based on the loss and weight of each branch network. For example, the loss of each branch network can be weighted and summed according to the weight of each branch network to train the network model as the loss of the entire network model. In this way, not only can the perception prediction algorithms of multiple branch networks be integrated into the same algorithm model, but the loss function of each branch network can also be comprehensively used to train the network model more fully, so as to obtain a multi-task perception model with high computational efficiency and prediction efficiency.

[0059] Figure 3The method flow of step S13 in the point cloud scene perception method according to an embodiment of the present invention is schematically shown. Figure 3 As shown, the method can be specifically implemented by comprising the following steps:

[0060] Step S31: inputting the bird's-eye view grid expression into the backbone network of the pre-trained network model, and obtaining a feature map integrating spatial information and temporal information from the backbone network;

[0061] Step S32: input the feature map into each branch network respectively, and obtain the branch prediction result output by the corresponding branch network from each branch network as the perception result.

[0062] Since the input part of the network model, that is, the bird's-eye view grid expression of the time series data, has been obtained in step S12, the bird's-eye view grid expression obtained in step S12 can be directly used as input and input into the backbone network of the network model in step S31. The bird's-eye view grid expression is processed by the spatiotemporal convolutional neural network to obtain a feature map that integrates spatial information and temporal information.

[0063] In step S32, the feature map outputted from step S31 is inputted into each branch network, so that each branch network is used to regress the feature map, and the branch prediction results corresponding to each branch network are obtained as the perception results of the corresponding branch network, and all the branch prediction results are used together as the perception results of the current task scene. Exemplarily, taking the branch network including a target detection branch network, a foreground segmentation branch network, a dynamic and static detection branch network, a motion prediction branch network, and an inter-frame displacement prediction branch network as an example, each branch network is used to regress the feature map, and the corresponding target detection results, foreground segmentation results, dynamic and static detection results, motion prediction results, and five branch prediction results of the relative displacement of the vehicle between the point cloud data of every two consecutive frames can be obtained. All of them are used as perception results, and a bird's-eye view grid expression based on time series data can be realized. Through a trained network model, five perception results can be obtained at the same time, and multi-task perception of the current point cloud scene can be realized.

[0064] The method provided by the embodiment of the present invention adopts the acquired continuously collected n-frame point cloud data as the basic information for obtaining the perception result at the current moment, so that the spatial information and time information can be fully considered when predicting the perception result. Compared with the perception algorithm of the prior art that performs perception prediction based on a single frame image as input, since the time series data including continuous n-frame point cloud data adopted by the present invention can represent more data information (that is, the feature expression is richer and includes spatial and temporal information at the same time), more accurate perception results can be obtained by performing perception prediction based on it, effectively improving the calculation accuracy of the perception algorithm, and more perception results of different forms can be obtained, which is beneficial to subsequent trajectory prediction, path planning and other operations, and improves the safety of autonomous driving.

[0065] Figure 4 The present invention schematically shows a point cloud scene sensing device according to an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0066] A time series data acquisition module 1 is used to acquire the time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data continuously collected up to the current moment, wherein n is a positive integer greater than 1;

[0067] A bird's-eye view rasterization module 2 is used to obtain a bird's-eye view raster expression of the time series data, wherein the bird's-eye view raster expression is represented by a bird's-eye view raster matrix corresponding to each frame of point cloud data in the time series data;

[0068] The perception prediction module 3 is used to obtain the perception result of the current point cloud scene at the current moment based on the bird's-eye view grid expression and the pre-trained network model unit.

[0069] Among them, the perception results include static perception results and dynamic perception results. The static perception results include at least one of the target detection results and the foreground segmentation results. The dynamic perception results include at least one of the dynamic and static detection results, the motion prediction results and the relative displacement between the n frames of point cloud data collected continuously.

[0070] It should be noted that the implementation process and implementation principle of the point cloud scene-based perception device of the embodiment of the present invention can be specifically referred to the corresponding description of the above-mentioned point cloud scene-based perception method embodiment, such as the method embodiment part for obtaining the time series data of the current point cloud scene, obtaining the bird's-eye view grid expression of the time series data, and obtaining the perception results, so they are not repeated here. Exemplarily, the point cloud scene-based perception device of the embodiment of the present invention can be any intelligent device with a processor, including but not limited to computers, smart phones, personal computers, robots, cloud servers, etc.

[0071] Figure 5A point cloud scene sensing device according to another embodiment of the present invention is schematically shown. Figure 5 As shown, in the device:

[0072] The pre-trained network model unit in the perception prediction module 3 includes a trunk network unit 31 for obtaining a feature map integrating spatial information and temporal information and at least one branch network unit 32 for obtaining a prediction result of a perception target;

[0073] The backbone network unit 31 takes the bird's-eye view grid expression as input and the feature map as output;

[0074] The branch network unit 32 takes a feature map integrating spatial information and temporal information as input and a branch prediction result as output;

[0075] The perception prediction module 3 is specifically used to input the bird's-eye view grid expression into the trunk network unit 31, and obtain a feature map that integrates spatial information and temporal information from the trunk network unit 31, and then input the feature map into each branch network unit 32 respectively, and obtain the branch prediction result output by the corresponding branch network from each branch network unit 32 as the perception result.

[0076] It should be noted that the implementation process and implementation principle of the point cloud scene-based perception device of the embodiment of the present invention can be specifically referred to the corresponding description of the above-mentioned point cloud scene-based perception method embodiment, such as the pre-trained network model content and the method for obtaining the perception result in the method embodiment, so they are not repeated here. Exemplarily, the point cloud scene-based perception device of the embodiment of the present invention can be any intelligent device with a processor, including but not limited to computers, smart phones, personal computers, robots, cloud servers, etc.

[0077] Figure 6 A mobile tool according to an embodiment of the present invention is schematically shown. Figure 6 As shown, the mobile tool includes:

[0078] Fuselage 41;

[0079] The point cloud scene-based perception device 42 is disposed on the fuselage 41 and is used to obtain the perception result of the current point cloud scene at the current moment.

[0080] The specific implementation process and implementation principle of the point cloud scene-based perception device 42 can be found in the corresponding description of the above embodiment, which will not be repeated here. It should be noted that the mobile tool in the embodiment of the present invention can be an unmanned cleaning vehicle, an unmanned cleaning machine, a cleaning robot, etc. with an automatic cleaning function.

[0081] Figure 7A mobile tool according to another embodiment of the present invention is schematically shown. Figure 7 As shown, the mobile tool includes:

[0082] A collection device 51, used to collect real-time point cloud data from the surrounding environment;

[0083] The controller 52 is used to obtain n frames of point cloud data continuously collected up to the current moment from the acquisition device 51 as time series data of the current point cloud scene, and obtain a bird's-eye view grid expression of the realization data, and obtain a perception result of the current point cloud scene at the current moment based on the bird's-eye view grid expression of the time series data and a pre-trained network model;

[0084] The planner 53 is used to plan and control the mobile tool according to the perception result output by the controller 52.

[0085] The acquisition device 51 may be a laser point cloud radar or other equipment installed on the autonomous driving vehicle, and the specific implementation process and implementation principle of the controller 52 may refer to the corresponding description of the above embodiment, which will not be repeated here. It should be noted that the mobile tool in the embodiment of the present invention may be an unmanned cleaning vehicle, an unmanned cleaning machine, a cleaning robot, etc. with an automatic cleaning function.

[0086] The “mobile tool” referred to in the embodiment of the present invention may be a vehicle with an L0-L5 autonomous driving technology level as established by the Society of Automotive Engineers International (SAE International) or the Chinese national standard “Automotive Driving Automation Classification”.

[0087] Exemplarily, the mobile tool may be a vehicle device or a robot device having the following various functions:

[0088] (1) Passenger-carrying function, such as family cars and buses;

[0089] (2) Cargo-carrying functions, such as ordinary trucks, van trucks, trailer trucks, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures, etc.;

[0090] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol cars, cranes, hoists, excavators, bulldozers, forklifts, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinklers, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;

[0091] (4) Entertainment functions, such as entertainment vehicles, amusement park self-driving devices, balance vehicles, etc.;

[0092] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.

[0093] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the point cloud scene-based perception method of any of the above embodiments of the present invention.

[0094] In some embodiments, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the point cloud scene-based perception method of any one of the above embodiments.

[0095] In some embodiments, an embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the point cloud scene-based perception method of any of the above embodiments.

[0096] In some embodiments, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the point cloud scene-based perception method of any of the above-mentioned embodiments is implemented.

[0097] Figure 8 is a schematic diagram of the hardware structure of an electronic device for executing a point cloud scene-based perception method provided by another embodiment of the present application, such as Figure 8 As shown, the device includes:

[0098] One or more processors 610 and memory 620, Figure 8 A processor 610 is taken as an example.

[0099] The device for executing the point cloud scene-based perception method may further include: an input device 630 and an output device 640 .

[0100] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0101] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the point cloud scene-based perception method in the embodiment of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, realizing the point cloud scene-based perception method in the above method embodiment.

[0102] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of a perception method based on a point cloud scene, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely disposed relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The input device 630 may receive input digital or character information and generate signals related to user settings and function control of the image processing device. The output device 640 may include a display device such as a display screen.

[0104] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the point cloud scene-based perception method in any of the above method embodiments is executed.

[0105] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0106] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0107] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0108] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0109] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0110] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0111] (5) Other electronic devices with data interaction functions.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. The perception method based on point cloud scene is characterized by: include: Acquire time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data collected continuously up to the current moment, wherein n is a positive integer greater than 1; Acquire a bird's-eye view grid expression of the time series data, wherein the bird's-eye view grid expression is represented by a bird's-eye view grid matrix corresponding one-to-one to each frame of point cloud data in the time series data; Based on the bird's-eye view grid expression and the pre-trained network model, the perception result of the current point cloud scene at the current moment is obtained.

2. The method according to claim 1, characterized in that The step of obtaining the bird's-eye view grid expression of the time series data comprises: Project each frame of point cloud data in the time series data onto the XY plane through raster encoding; According to the projection results of each frame of point cloud data, the bird's-eye view grid matrix corresponding to the corresponding frame of point cloud data is obtained.

3. The method according to claim 1, characterized in that: The pre-trained network model is a multi-task perception model that takes a bird's-eye view grid expression of time series data as input and takes at least two perception results as output.

4. The method according to claim 3, characterized in that The perception results include static perception results and dynamic perception results. The static perception results include at least one of target detection results and foreground segmentation results. The dynamic perception results include dynamic and static detection results, motion prediction results, and at least one of the relative displacement of the vehicle between point cloud data of every two consecutive frames in the time series data.

5. The method according to claim 1, characterized in that The pre-trained network model includes a backbone network for feature fusion of spatial information and temporal information based on time series data. The backbone network is a spatiotemporal convolutional neural network that takes a bird's-eye view grid expression of time series data as input and a feature map as output.

6. The method according to claim 5, characterized in that The pre-trained network model also includes at least one branch network, wherein each branch network takes the output of the trunk network as input, and each branch network outputs a different type of branch prediction result.

7. The method according to claim 6, characterized in that The loss of the network model is determined by the branch loss corresponding to each branch network and the weight assigned to each branch network.

8. The method according to claim 7, characterized in that The branch network includes a target detection branch network with a target detection result as an output branch prediction result, a foreground segmentation branch network with a foreground segmentation result as an output branch prediction result, a dynamic and static detection branch network with a dynamic and static detection result as an output branch prediction result, a motion prediction branch network with a motion prediction result as an output branch prediction result, and an inter-frame displacement prediction branch network with a relative displacement of the vehicle between point cloud data of every two consecutive frames in the time series data as an output branch prediction result.

9. The method according to any one of claims 6 to 8, characterized in that: The obtaining of the perception result of the current point cloud scene at the current moment based on the bird's-eye view grid expression and the pre-trained network model includes: Inputting the bird's-eye view grid expression into the backbone network of the pre-trained network model, and obtaining a feature map integrating spatial information and temporal information from the backbone network; The feature maps are input into each branch network respectively, and the branch prediction results output by the corresponding branch network are obtained from each branch network as the perception results.

10. A point cloud scene-based perception device, characterized in that: include: A time series data acquisition module, used to acquire time series data of the current point cloud scene, wherein the time series data includes n frames of point cloud data continuously collected up to the current moment, wherein n is a positive integer greater than 1; A bird's-eye view rasterization module, used to obtain a bird's-eye view raster expression of the time series data, wherein the bird's-eye view raster expression is represented by a bird's-eye view raster matrix corresponding to each frame of point cloud data in the time series data; The perception prediction module is used to obtain the perception result based on the bird's-eye view grid expression and the pre-trained network model unit.

11. An electronic device, characterized in that: include: A memory for storing executable instructions; as well as A processor, configured to execute executable instructions stored in a memory, wherein the executable instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

12. A mobile tool, characterized in that The mobile tool includes: the point cloud scene-based perception device described in claim 9 or the electronic device described in claim 11.

13. A mobile tool, characterized in that The mobile tools include: A collection device for collecting real-time point cloud data from the surrounding environment; A controller, used to obtain n frames of point cloud data continuously collected up to the current moment from the acquisition device as time series data of the current point cloud scene, and obtain a bird's-eye view grid expression of the time series data, and obtain a perception result of the current point cloud scene at the current moment based on the bird's-eye view grid expression of the time series data and a pre-trained network model, wherein n is a positive integer greater than 1, and the bird's-eye view grid expression is represented by a bird's-eye view grid matrix corresponding to each frame of point cloud data in the time series data; A planner is used to plan and control the mobile tool according to the perception results output by the controller.

14. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.

15. A computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images