Local road information generation method and device based on perception data, computing equipment, vehicle, storage medium and computer program product

Abstract

The invention discloses a sensing data-based local road information generation method and device, computing equipment, a vehicle, a storage medium and a computer program product, and the method comprises the steps: obtaining first data information which comprises sensing lane line data; based on the first data information and a pre-trained lane information model, first lane information is obtained from the lane information model, and the first lane information comprises lane information and a lane topological relation; and generating and outputting second lane information according to the first lane information, wherein the second lane information comprises a lane boundary line. According to the method provided by the embodiment of the invention, the lane information can be obtained based on the sensing lane line data obtained by processing the data obtained by the sensing module and the pre-trained lane information model, so that the road information can be generated without depending on data in a high-precision map, and the cost is effectively reduced; and road condition changes can be quickly adapted, and the driving safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, computing equipment, vehicle, storage medium and computer program product for generating local road information based on perception data. Background Art

[0002] With the rapid development of science and technology, autonomous driving technology is increasingly being used. At present, most autonomous driving technologies rely on obtaining road information such as lane information and lane topology from high-precision maps (HDMaps). However, HDMaps have the disadvantages of high construction and maintenance costs, slow updates, high production costs, large data volumes, and strict regulatory supervision. Therefore, the cost of relying on HDMaps to obtain road information is very high. In addition, since HDMaps are updated slowly, in actual applications, there is a situation where the data in the HDMaps is inconsistent with the actual conditions of the environment in which the autonomous driving vehicle is currently located. When the data in the HDMaps is inconsistent with the actual conditions of the environment in which the autonomous driving vehicle is currently located, relying entirely on the road information provided by the HDMaps will cause the autonomous driving system to make decisions and planning errors, bringing safety risks. Summary of the invention

[0003] Embodiments of the present invention provide a method, apparatus, computing device, vehicle, storage medium, and computer program product for generating local road information based on perception data, so as to solve the problems in the prior art of relying on data in high-precision maps to obtain road information such as lane information and lane topology relationships, which results in high costs and driving safety risks such as decision-making and planning.

[0004] In a first aspect, an embodiment of the present invention provides a method for generating local road information based on perception data, comprising:

[0005] Acquire first data information, wherein the first data information includes sensed lane line data;

[0006] Based on the first data information and the pre-trained lane information model, obtaining first lane information from the lane information model, where the first lane information includes lane information and lane topology relationship;

[0007] A second lane information output is generated according to the first lane information, wherein the second lane information includes a lane boundary line.

[0008] In a second aspect, an embodiment of the present invention provides a device for generating local road information based on perception data, including:

[0009] A first data information acquisition module, used to acquire first data information, wherein the first data information includes sensed lane line data;

[0010] A first lane information acquisition module, configured to acquire first lane information from a lane information model based on first data information and a pre-trained lane information model, wherein the first lane information includes lane information and a lane topology relationship;

[0011] The second lane information acquisition module is used to generate second lane information output according to the first lane information, where the second lane information includes a lane boundary line.

[0012] In a third aspect, an embodiment of the present invention provides a computing device, including:

[0013] A memory for storing executable instructions; and

[0014] A processor is used to execute executable instructions stored in a memory, wherein the executable instructions implement the steps of the method of the first aspect mentioned above when executed by the processor.

[0015] In a fourth aspect, an embodiment of the present invention provides a mobile tool, comprising the local road information generation device based on perception data of the second aspect or the computing device of the third aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in the first aspect.

[0017] In a sixth aspect, an embodiment of the present invention provides a computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the steps of the method of the first aspect above.

[0018] The method provided by the embodiment of the present invention obtains the first lane information by processing the perceived lane line data obtained by the perception module and the pre-trained lane information model, and uses the first lane information to generate the second lane information including the lane boundary line, thereby realizing the real-time generation of local lane information around the autonomous driving vehicle directly based on the perception data information provided by the perception technology of autonomous driving, and thus no longer relying on the data in the high-precision map, and generating complete road information, forming lane boundary lines, lane lines and other data. Compared with the traditional method of relying on high-precision maps to generate road information, it can effectively reduce costs, and at the same time can quickly adapt to changes in road conditions and improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a method for generating local road information based on perception data according to an embodiment of the present invention;

[0021] Figure 2 It is a flowchart of a method for training a lane information model in a method for generating local road information based on perception data according to an embodiment of the present invention;

[0022] Figure 3 A training flow chart of a lane information model in a method for generating local road information based on perception data according to an embodiment of the present invention;

[0023] Figure 4 A flowchart of a method for generating second lane information in a method for generating local road information based on perception data according to an embodiment of the present invention;

[0024] Figure 5 A flowchart of generating second lane information in a method for generating local road information based on perception data according to an embodiment of the present invention

[0025] Figure 6 This is an overall method flow chart of a method for generating local road information based on perception data according to an embodiment of the present invention;

[0026] Figure 7 It is a principle block diagram of a device for generating local road information based on perception data according to an embodiment of the present invention;

[0027] Figure 8 It is a principle block diagram of a device for generating local road information based on perception data according to another embodiment of the present invention;

[0028] Fig. 9 A functional block diagram of a computing device according to an embodiment of the present invention;

[0029] Fig.10 This is a principle block diagram of a mobile tool according to an embodiment of the present invention;

[0030] Fig.11 It is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0033] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0034] In the present invention, "module", "device", "system" and the like refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, an element can be, but is not limited to, a process, a processor, an object, an executable element, an execution thread, a program and / or a computer running on a processor. In addition, an application or script program running on a server, a server can all be an element. One or more elements can be in an execution process and / or thread, and an element can be localized on a computer and / or distributed between two or more computers, and can be operated by various computer-readable media. An element can also communicate through local and / or remote processes according to a signal with one or more data packets, for example, a signal from a data that interacts with another element in a local system, a distributed system, and / or a network on the Internet through a signal to interact with other systems.

[0035] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include" and "comprise" include not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such processes, methods, articles or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment that includes the elements.

[0036] The local road information generation method based on perception data in the embodiment of the present invention can be applied to the local road information generation device based on perception data, so that the vehicle can use the local road information generation device based on perception data to generate road information based on the data obtained by the perception module, such as lane lines, lane boundary lines, etc., to avoid the defects and problems caused by the reliance on high-precision maps. These local road information generation devices based on perception data include, for example, but are not limited to smart phones, smart tablets, personal PCs, computers, cloud servers, etc. In particular, the local road information generation method based on perception data in the embodiment of the present invention can also be applied to smart devices with autonomous driving functions (or mobile platforms provided with autonomous driving functions), such as unmanned vehicles, robots or autonomous driving vehicles, etc., and the present invention is not limited to this.

[0037] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0038] Figure 1 The process of a method for generating local road information based on perception data according to an embodiment of the present invention is schematically shown. The method is suitable for analyzing and processing data obtained by a perception module received by a local road information generating device based on perception data, such as a smart phone, a personal computer, a cloud server, etc., to obtain road information near the local road information generating device based on perception data. The method is also suitable for generating nearby road information on movable platforms with autonomous driving functions, such as autonomous driving vehicles, unmanned vehicles, robots, etc., and enabling these movable platforms to generate road information near their surroundings for use by subsequent other functional modules. The execution subject can be a processor of a local road information generating device based on perception data, or a processor of a movable platform such as an unmanned vehicle, a robot, an autonomous driving vehicle, etc., and the embodiments of the present invention are not limited to this. Reference Figure 1 As shown, the method comprises the following steps:

[0039] Step S11: Acquire first data information, wherein the first data information includes sensed lane line data;

[0040] Step S12: based on the first data information and the pre-trained lane information model, obtaining first lane information from the lane information model, where the first lane information includes lane information and lane topology relationship;

[0041] Step S13: Generate second lane information output according to the first lane information, where the second lane information includes a lane boundary line.

[0042] In step S11, the first data information includes perceived lane line data, which is data information for representing the perceived lane line obtained after algorithm processing such as object detection and classification based on the sensor data acquired by the perception module, and the lane line may include lane dividers, road edges, stop lines, crosswalks, etc. In specific applications, the perception module and the algorithm processing module for object detection and classification are both known front modules, so in the embodiment of the present invention, the perceived lane line data can be directly acquired from the front module with these functions, and subsequent processing is performed based on the acquired perceived lane line data to achieve real-time generation of road information around the autonomous driving vehicle based on perception technology, avoiding dependence on high-precision maps. As a possible implementation, the sensor data acquired by the perception module at least includes image data of the lane line. In this implementation, when the perception module acquires the sensor data, it can acquire it through a camera set on the autonomous driving vehicle. The camera can recognize all lanes and identification elements, especially lane lines drawn in various colors, so the camera set on the autonomous driving vehicle can effectively acquire image data containing lane lines. As another possible implementation, the sensor data acquired by the perception module may also include 3D point cloud data. In this implementation, when acquiring sensor data, the perception module may also acquire it through a laser radar set on the autonomous driving vehicle. The laser radar has a good detection effect on road elements with 3D size and height, such as road boundaries, curbs, isolation belts, tunnel walls, etc., so it can better detect road signs around the road to obtain 3D point cloud data. After obtaining the sensor data, the perceived lane line data can be obtained after processing. Exemplarily, after the perception module acquires the sensor data, it can detect the required target objects (such as lane lines) and convert the position information of these objects to the BEV coordinate system. The data form of the perceived lane line data obtained after processing the sensor data can be one of the following: 1) Dense discrete point sequence: a discrete point sequence with close intervals (for example, 0.5 meters) to express a continuous curve, and each point in the curve is a (x, y) coordinate value; 2) Curve equation: an equation of y about x to express the curve; 3) Line segment sequence: an approximate curve expression composed of multiple line segments. Therefore, in the embodiment of the present invention, the lane line data sensed by the front module may also be in any one of the above three forms or a combination of two or more thereof.

[0043] As a possible implementation, the first data information may also include perceived road sign data, which is data information for identifying perceived road signs after algorithm processing such as object detection and classification based on the sensor data acquired by the perception module. Road signs may include signs such as traffic lights and speed limit signs standing beside the road, signs such as turn signs drawn on the road, etc. Similarly, the perceived road sign data can be obtained from the front module. Specifically, for the front module, the sensor data for obtaining the perceived road sign may be image data of the road sign acquired by a camera set on the autonomous driving vehicle, and may also include 3D point cloud data acquired by a laser radar set on the autonomous driving vehicle. The data information of the perceived road sign data obtained after processing the sensor data may include the following: 1) Sign type: including but not limited to, for example, deceleration signs, turn signs, fork signs, speed limit signs, traffic lights, etc.; 2) Sign position: x and y coordinate values ​​in the BEV coordinate system; 3) Sign size: generally the size occupied in the x and y directions in the BEV coordinate system, and a minimum size can be agreed to express signs that are too small. Therefore, in the embodiment of the present invention, the perceived road sign data obtained from the front module may also be in any one of the above three forms or a combination of two or more thereof.

[0044] Step S12 is a step of acquiring the first lane information based on the first data information and a pre-trained lane information model. Among them, the pre-trained lane information model in the embodiment of the present invention is specifically an algorithm model for generating lane information data in real time based on perception data, which can be a machine learning model with the first data information as input and the first lane information as output. Considering that the data format of the first data information obtained from the front module may be more than one, and the data format expression of the input part of the lane information model to the lane information model needs to be unified, as a preferred implementation, the first data information can be pre-processed first to obtain the first data information in a specific data format that can be used as the input part of the lane information model, thereby reducing the training cost and training complexity of the lane information model and improving the accuracy of the trained lane information module. Take the example of a trained lane information model with the first data information in a specific expression form as the input part and the first lane information as the output part, Figure 2 The method flow of step S12 in the method for generating local road information based on perception data according to an embodiment of the present invention is schematically shown. Figure 2 As shown, step S12 can be specifically implemented as including the following steps:

[0045] Step S21: preprocessing the first data information to obtain second data information, wherein the second data information includes the first data information in a specific expression form;

[0046] Step S22: inputting the second data information into a pre-trained lane information model to obtain the first lane information from the lane information model.

[0047] Step S21 is a step of preprocessing the first data information obtained in step S11, and preprocessing the first data information to obtain second data information that can be used as an input of the lane information model. Exemplarily, the first data information in a specific expression form that can be used as the input part of the lane information model can be first data information in a raster expression form, or can be first data information in a vector expression form. Accordingly, the preprocessing of the first data information can include two methods, one of which is to process the first data information format into a raster image expression method, and the other is to process the first data information format into a vector set expression method.

[0048] Taking the expression of converting the first data information into a raster image as an example, the preprocessing of the first data information can be implemented by performing rasterization data processing on the first data information to obtain second data information including a raster expression corresponding to the first data information. Since the first data information may contain both the perceived lane line data and the perceived road sign data, the data forms of the perceived lane line data and the perceived road sign data are different, and the data form of the perceived lane line data may be one of a dense discrete point sequence, a curve equation, and a line segment sequence, the rasterization data processing method is different according to the different data forms of the first data information.

[0049] Wherein, when the first data information only includes the perceived lane line data, the second data information generated by preprocessing includes the first grid expression corresponding to the perceived lane line data. When the data form of the perceived lane line data is a dense discrete point sequence, the preprocessing of the first data information can be exemplarily implemented as including first rasterizing each discrete point according to the set map resolution, and converting the coordinate value of each grid point into the overall image coordinate, and then connecting each rasterized discrete point in sequence to generate a polyline, and finally drawing the polyline on the picture to generate a raster picture. When the data form of the perceived lane line data is a curve equation, the preprocessing thereof can be exemplarily implemented as including first calculating a fixed number of waypoints at equal intervals according to the curve equation, and then rasterizing each calculated waypoint, and finally connecting the rasterized adjacent waypoints in sequence to generate a polyline, and finally drawing the polyline on the picture to generate a raster picture. When the data form of the perceived lane line data is a line segment sequence, the preprocessing thereof can be exemplarily implemented as including directly drawing each line segment on the picture according to the line segment sequence to generate a raster picture.

[0050] When the first data information also includes perceived road sign data, the second data information generated by preprocessing includes the perceived lane line data and the second grid expression corresponding to the perceived road sign data. For the perceived road sign data, the perceived road sign data contains the sign type, sign position and sign size, and the preprocessing thereof can be implemented as including first converting the sign size into a picture scale according to the sign position and sign size, and the set map resolution, and then generating a corresponding size rectangle of the picture scale, and then directly drawing the generated size rectangle as a sign on the picture to generate a raster picture. Among them, when the perceived lane line data and the perceived road sign data are rasterized to generate a raster picture, they can be generated on the same raster picture. At this time, the preprocessing order of the perceived lane line data and the perceived road sign data can be to first preprocess the perceived lane line data to obtain a raster picture including a polyline generated based on the perceived lane line data, and then continue to preprocess the perceived road sign on the obtained raster picture to finally generate a raster picture including both a polyline generated based on the perceived lane line data and a sign generated based on the perceived road sign data as the second data information.

[0051] Taking the expression of converting the first data information into a vector set as an example, the preprocessing of the first data information can be implemented as vectorized data processing of the first data information to obtain second data information including a vector expression corresponding to the first data information. Since the first data information may contain both perceived lane line data and perceived road sign data, the data forms of the perceived lane line data and the perceived road sign data are not the same, and the data form of the perceived lane line data may also be one of a dense discrete point sequence, a curve equation, and a line segment sequence, the vectorized data processing method is different for different data forms of the first data information. The vector expression obtained by converting the perceived lane line is represented by a first vector set composed of at least one vector generated based on the perceived lane line data, and the vector expression obtained by converting the perceived road representation data is represented by a second vector set composed of at least one vector generated based on the perceived road sign data.

[0052] Among them, for the first vector set, it is composed of at least one vector generated based on the perceived lane line data. When the data form of the perceived lane line data is a dense discrete point sequence, its preprocessing can be exemplarily implemented as including first sampling the perceived lane line data for a fixed number of times to generate a sparser sampling point sequence, then calculating vectors based on adjacent sampling points, and finally generating all calculated vectors into a vector set according to the calculation results. When the data form of the perceived lane line data is a curve equation, its preprocessing can be exemplarily implemented as including first calculating a fixed number of waypoints at equal intervals according to the curve equation, then calculating vectors for adjacent waypoints, and finally generating all calculated vectors into a vector set according to the calculation results. When the perceived lane line data is a line segment sequence, its preprocessing can be exemplarily implemented as including calculating vectors based on the start and end points of each line segment in the perceived lane line data, and generating all calculated vectors into a vector set.

[0053] The second vector set is composed of at least one vector generated based on the perceived road sign data. For the perceived road sign data, the perceived road sign data contains the sign type, sign position and sign size, and the preprocessing thereof can be exemplarily implemented as first generating a rectangle of corresponding size according to the sign position and the sign size, then calculating a vector for each side of the rectangle, and generating the vectors calculated for all sides of the rectangle into a vector set.

[0054] The above are two of the methods for unifying the data format of the first data information. In addition to unifying the data format of the first data information into a data expression of a raster image or a vector set, the data format of the first data information can also be unified into a data expression of a directed graph. The specific method can refer to the relevant steps of the rasterization and vectorization conversion in the previous text, and no further explanation is given here. As for the preprocessing of the first data information, in addition to unifying the data format of the first data information, it can also include filtering and calibration of some elements. The specific elements that need to be filtered and calibrated can be designed according to the actual situation, and no further explanation is given here.

[0055] After preprocessing the first data information in step S21 to obtain the second data information, the second data information can be input into a pre-trained lane information model according to step S22 to obtain the first lane information, wherein the first lane information includes lane information and lane topology relationship, the lane information is data information used to represent the lane, which may include the lane centerline, lane width, lane boundary type, etc., and the lane topology relationship is the topological relationship between each lane line.

[0056] The lane information model is a pre-trained model. Through the lane information model, the first lane information that can represent the lane information and topological relationship of the current perception range can be obtained based on the lane line data obtained by the perception technology. Continuing with the example that the trained lane information model takes the first data information in a specific expression form as the input part and the first lane information as the output part, Figure 3 The process of the training method of the lane information model in the method for generating local road information based on perception data according to an embodiment of the present invention is schematically shown. Figure 3 As shown, the training method of the lane information model can be specifically implemented by including the following steps:

[0057] Step S31: acquiring training data information, wherein the first training data information includes first sample data information and true lane information, wherein the first sample data information includes sample-perceived lane line data, and the true lane information includes real lane information and real lane topology relationship extracted from the position covered by the sample-perceived lane line data in the high-precision map;

[0058] Step S32: preprocessing the first sample data information to obtain second sample data information, wherein the second sample data information includes the first sample data information in a specific expression form;

[0059] Step S33: inputting the second sample data information into the selected machine learning model to obtain training lane information corresponding to the first sample data information from the machine learning model, and performing loss calculation based on the training lane information and the true value lane information to obtain an adjustment coefficient of the machine learning model;

[0060] Step S34: iteratively train and adjust the machine learning model according to the obtained adjustment coefficient to obtain the lane information model.

[0061] The training data information obtained in step S31 is a sample set used to train the training model. Specifically, the training data information includes first sample data information and true lane information. The first sample data information includes sample perceived lane line data. The sample perceived lane line data is perceived lane line data used as a training data sample, and has the same source and data format as the perceived lane line data in the first data information. It is the perceived data obtained after processing the sensor data obtained based on the perception module. Therefore, the relevant description thereof can refer to the relevant description of the perceived lane line data in the previous text. The true lane information corresponds to the location of the first sample data information as the input of the model training in the high-precision map. It is the real lane information extracted from the location covered by the sample perceived lane line data in the high-precision map. In the process of training the machine learning model, the real road information is used to supervise the learning process of the model, so that the model can learn the correct connection between the lane topology and the lane line data, thereby improving the training effect of the model. Specifically, the true lane information includes the real lane information and the real lane topology relationship corresponding to the first sample data information. The real lane information may include, for example, the lane centerline coordinates, lane width, and lane boundary type, or may include the lane centerline coordinates, lane width, lane boundary type, and lane direction. The lane direction may be obtained according to the order of the waypoints in the lane centerline, and thus may not be extracted separately but may be determined based on the extracted lane centerline.

[0062] In some embodiments, the first sample data information may also include sample perceived road sign data. The sample perceived road sign data is the same as the perceived road sign data, which is the perceived data acquired based on the perception module. For details, please refer to the relevant description of the perceived road sign data in the previous text.

[0063] Step S32 is a step of preprocessing the first sample data information in the training data information to obtain the second sample data information used as the input of the model to be trained. Wherein, corresponding to the expression form of the first data information corresponding to the second data information, the format of the second sample data information can also be a raster image format or a vector set format. Regarding the method of preprocessing the first sample data information, reference can be made to the relevant description of preprocessing the first data information in the previous text, and no further description will be given here.

[0064] Step S33 is a step of training the model to be trained based on the true lane information in the training data information and the second sample data information obtained by preprocessing the first sample data information. The model to be trained is a machine learning model constructed to generate lane topology based on input perception information, which can receive data in a corresponding form and output information that can be used to construct lanes around the vehicle and their topological relationships. The constructed machine learning model includes but is not limited to decision tree models, linear models, nonlinear models, neural network models and other forms.

[0065] When training the machine learning model, the second sample data information is first input into the machine learning model, and then the training lane information corresponding to the first sample data information can be obtained from the machine learning model. It can be understood that the training lane information includes the sample lane information and the sample lane topological relationship corresponding to the first sample data information. Then, the loss calculation is performed based on the obtained training lane information and the true lane information to obtain the adjustment coefficient of the machine learning model. When performing the loss calculation, the obtained training lane information and the true lane information need to be input into multiple corresponding loss functions for loss calculation. The loss function can calculate the penalty and adjustment coefficient of the machine learning model based on the error between the training lane information output by the machine learning model and the true lane information. These loss functions can include lane centerline position loss function, lane width loss function, lane boundary line type loss function, lane association loss function and lane direction loss function.

[0066] Step S34 is to perform iterative training and adjustment on the machine learning model based on the adjustment coefficient obtained in step S33. After multiple iterative trainings, the training lane information output by the machine learning model can be closer to the true lane information. Specifically, after the adjustment coefficient is obtained, it is transmitted back to the machine learning model in the form of a gradient, so that the machine learning model adjusts its own parameters and reduces the error between the model output and the true value, so as to achieve the effect of model training.

[0067] As a possible implementation, the training data information may also include auxiliary training data, which is simulated perception lane data obtained by extracting relevant parts of the high-precision map and distorting them. The simulated perception lane data may include extracting lane lines at positions corresponding to the sample perception lane line data in the high-precision map, distorting them to obtain simulated perception lane line data, and extracting road signs at positions corresponding to the sample perception road sign data in the high-precision map, distorting them to obtain simulated perception road sign data. By extracting and distorting the corresponding parts of the high-precision map terminal, the perception data obtained by the perception module can be imitated, so that when the amount of collected perception data is insufficient or when supplementary scene data is required, the sample set can be supplemented, thereby enhancing the training effect of the model. The simulated perception lane data obtained by the distortion processing can be pre-processed data, that is, the data format is the data format for inputting the machine learning model. Therefore, when the machine learning model is trained using the auxiliary training data, it can be processed in the same way as the second sample data information to obtain the adjustment coefficient of the machine learning model, so as to iteratively train and adjust the machine learning model. It should be noted that the auxiliary training data is only used when training the machine learning model. When the trained lane information model is used, the auxiliary training data is not added as the input of the lane information model.

[0068] Figure 4 The training process of the lane information model in the method for generating local road information based on perception data according to an embodiment of the present invention is schematically shown. Figure 4 As shown in the figure, the second sample data information and true lane information are obtained after preprocessing the lane line expression, road sign expression and high-precision map lane information extraction representation. By inputting the second sample data information and the true lane information into the machine learning model, the machine learning model generates training lane information according to the second sample data information, and obtains the adjustment coefficient through the loss function, and then the adjustment coefficient gradient is back-transferred to the machine learning model. Through repeated iterations, the training of the machine learning model is completed to obtain a pre-trained lane information model.

[0069] Step S13 is a step of generating second lane information including lane boundary lines according to the first lane information output by the lane information model. The first lane information includes lane information and lane topology. Since the first lane information output by the lane information model is data information used to represent the lanes around the vehicle and their topology, as mentioned above, it includes lane information and lane topology such as lane centerline, lane width, lane boundary type, lane direction, etc., and does not directly output lane boundaries, it is necessary to perform post-processing to obtain the lane boundary lines of the real lanes that can be used by subsequent algorithm modules. Figure 5The method flow of generating the second lane information in the method of generating local road information based on perception data according to an embodiment of the present invention is schematically shown. Figure 5 As shown, the method can be specifically implemented by comprising the following steps:

[0070] Step S41: generating a virtual lane boundary according to the lane information in the first lane information;

[0071] Step S42: Bind the virtual lane boundary and the lane line detected by the perception module to generate a lane boundary line.

[0072] In step S41, since the lane information includes the lane centerline, lane width, lane boundary type, etc., a virtual lane boundary corresponding to the lane information can be generated based on the lane centerline and lane width in the lane information. Specifically, since the lane centerline is represented by multiple lane center line points, two boundary lines of each lane can be generated according to each lane center line point and lane width as the virtual lane boundary.

[0073] In step S42, since the virtual lane boundary generated in step S41 is not a line existing in the real world, it is necessary to bind the virtual lane boundary and the lane line detected by the perception module, and replace the virtual lane boundary with the real lane line to form a real lane boundary line that can be directly used by the subsequent algorithm module. Specifically, the virtual lane boundary and the lane line detected by the perception module are bound, and the virtual lane boundary with the real lane line bag body can be implemented as follows: first, the point set of the generated virtual lane boundary and the point set of the perception lane line obtained in step S11 are measured to determine the perception lane line closest to each virtual lane boundary; then the perception lane line with the closest distance is used to replace the corresponding virtual lane boundary, so as to complete the replacement of the virtual lane boundary with the real lane line to form a lane boundary line. Therefore, it is also possible to associate and bind the real lane line obtained by perception with the corresponding lane center line in the first lane information, so that the subsequent algorithm module can directly use the real lane line obtained by perception and the lane center line associated and bound with the real lane line determined by the lane information model to make driving decisions and planning, avoiding dependence on high-precision maps.

[0074] As a possible implementation, the second lane information generated in step S13 also includes a lane topology verification result. The trained lane information model can provide multiple lane information and the relative position relationship between these lanes (i.e., lane topology relationship), such as the successor lane, the left neighbor lane, etc. However, there may be inconsistencies between the lane position and the topological relationship output by the lane information model, so it is necessary to perform a rough check on the lane topology relationship based on the lane position. The purpose of the rough check is to check whether there are obvious errors in the lane topology relationship based on the lane position and the starting and ending points of the lane. For example, whether the end coordinates of a lane and the starting coordinates of its successor lane are within a certain threshold, if it is greater than a certain threshold, the topology relationship is considered to be wrong; if there is an error, the rough check step marks the erroneous data, wherein the mark represents the lane topology verification result, and the verification result can be used as a basis for whether the subsequent algorithm module needs to trust or correct the lane topology relationship, so that the obtained second data information has more reference value.

[0075] Figure 6 The overall method flow of the method for generating local road information based on perception data according to an embodiment of the present invention is schematically shown. Figure 6 As shown, firstly, based on the real-time perception data acquired by the perception module, the first data information is obtained, that is, Figure 6 The lane line expression and road sign expression in the first box on the left are then preprocessed on the first data information, that is, the data expression form of the first data information is converted to obtain the second data information that can be used as the input of the lane information model, that is, Figure 6 The lane line expression and road sign expression in the second box on the left. Then the second data information is used as the input of the lane information model to generate real-time first lane information based on the real-time perception data. The first lane information includes lane information and lane topology. Finally, the first lane information is post-processed to generate second lane information including lane boundary lines and lane topology verification results, completing the real-time generation and construction of local lane topology.

[0076] The method of the embodiment of the present invention can generate and construct the local lane information and lane topology around the vehicle in real time through the perception data perceived in real time by pre-training the above lane information model based on perception and deploying it on the vehicle, which not only avoids the dependence on high-precision maps and reduces the real-time construction cost of lane information and topology, but also can improve driving safety based on real-time perception of road conditions. Therefore, in a specific application, the method for generating local road information based on perception data provided by the embodiment of the present invention obtains the second data information for inputting the lane information model by pre-processing the perception lane line data obtained by perception, obtains the first lane information by using the lane information model, and generates the second lane information including the lane boundary line by using the first lane information, thereby generating the lane boundary line without relying on the data in the high-precision map. Compared with the traditional method of generating lane boundary lines relying on high-precision maps, it can effectively reduce the cost and quickly adapt to changes in road conditions.

[0077] Figure 7 The present invention schematically shows a device for generating local road information based on perception data according to an embodiment of the present invention. Figure 7 As shown, the local road information generation device based on perception data includes:

[0078] A first data information acquisition module 1, used to acquire first data information, wherein the first data information includes sensed lane line data;

[0079] A first lane information acquisition module 2, configured to acquire first lane information from a lane information model based on first data information and a pre-trained lane information model, wherein the first lane information includes lane information and a lane topology relationship;

[0080] The second lane information generating module 3 is used to generate second lane information output according to the first lane information, where the second lane information includes a lane boundary line.

[0081] Among them, the lane information model is an algorithm model used to generate lane information data in real time based on perception data, which can be a machine learning model that takes the first data information or its specific expression form as input and takes the first lane information as output.

[0082] It should be noted that the implementation process and implementation principle of the local road information generation device based on perception data in the embodiment of the present invention can be specifically referred to the corresponding description of the above-mentioned local road information generation method based on perception data, such as the corresponding processing of the acquisition of the first data information, the training method of the lane information model, and the method of generating the second lane information in the method embodiment, so they are not repeated here. Exemplarily, the local road information generation device based on perception data in the embodiment of the present invention can be any intelligent device with a processor, including but not limited to computers, smart phones, personal computers, robots, cloud servers, etc.

[0083] Figure 8 A device for generating local road information based on perception data according to another embodiment of the present invention is schematically shown. Figure 8 As shown, in this embodiment, the first lane information acquisition module 2 of the local road information generation device based on perception data includes:

[0084] A preprocessing unit 21, configured to preprocess the first data information to obtain second data information, wherein the second data information includes the first data information in a specific expression form;

[0085] The lane information model unit 22 is used to input the second data information into a pre-trained lane information model to obtain the first lane information from the lane information model.

[0086] It should be noted that the implementation process and implementation principle of the local road information generation device based on perception data in the embodiment of the present invention can be specifically referred to the corresponding description of the local road information generation method based on perception data in the above-mentioned embodiment, such as the corresponding processing of the preprocessing of the first data information and the training method of the lane information model in the method embodiment, so they are not repeated here. Exemplarily, the local road information generation device based on perception data in the embodiment of the present invention can be any intelligent device with a processor, including but not limited to computers, smart phones, personal computers, robots, cloud servers, etc.

[0087] Fig. 9 A computing device according to an embodiment of the present invention is schematically shown. Fig. 9 As shown, the computing device includes:

[0088] A memory 41, used for storing executable instructions; and

[0089] The processor 42 is used to execute executable instructions stored in the memory, and the executable instructions, when executed by the processor, implement the steps of the above-mentioned method for generating local road information based on perception data.

[0090] It should be noted that the implementation process and implementation principle of the computing device of an embodiment of the present invention when executing the local road information generation method based on perception data can be specifically referred to the corresponding description of the above-mentioned local road information generation method based on perception data embodiment, such as the corresponding processing of preprocessing the first data information, the training method of the lane information model, and the method of generating the second lane information in the method embodiment part, so they will not be repeated here.

[0091] Fig.10 A mobile tool according to an embodiment of the present invention is schematically shown. Fig.10 As shown, the mobile tool includes:

[0092] fuselage 51;

[0093] The above-mentioned local road information generating device 52 based on perception data or the above-mentioned computing device.

[0094] Among them, the specific implementation process and implementation principle of the local road information generation device based on perception data can be found in the corresponding description of the above embodiment, which will not be repeated here.

[0095] The "mobile tool" referred to in the embodiment of the present invention can be a vehicle with L0-L5 autonomous driving technology level established by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Automotive Driving Automation Classification".

[0096] Exemplarily, the mobile tool may be a vehicle device or a robot device having the following various functions:

[0097] (1) Passenger-carrying function, such as family cars and buses;

[0098] (2) Cargo-carrying functions, such as ordinary trucks, van trucks, trailer trucks, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures, etc.;

[0099] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol cars, cranes, hoists, excavators, bulldozers, forklifts, road rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, water sprinklers, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;

[0100] (4) Entertainment functions, such as entertainment vehicles, amusement park self-driving devices, balance vehicles, etc.;

[0101] (5) Special rescue functions, such as fire trucks, ambulances, power repair vehicles, engineering rescue vehicles, etc.

[0102] In some embodiments, an embodiment of the present invention provides a non-volatile computer-readable storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the method for generating local road information based on perception data according to any of the above embodiments of the present invention.

[0103] In some embodiments, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the method for generating local road information based on perception data of any one of the above embodiments.

[0104] In some embodiments, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the local road information generation method based on perception data of any of the above embodiments.

[0105] In some embodiments, an embodiment of the present invention further provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the method for generating local road information based on perception data of any of the above embodiments is implemented.

[0106] Fig.11 is a schematic diagram of the hardware structure of an electronic device for executing a method for generating local road information based on perception data provided by another embodiment of the present invention, such as Fig.11 As shown, the device includes:

[0107] One or more processors 610 and memory 620, Fig.11 A processor 610 is taken as an example.

[0108] The device for executing the method for generating local road information based on perception data may further include: an input device 630 and an output device 640 .

[0109] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0110] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method for generating local road information based on perception data in the embodiment of the present invention. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, the method for generating local road information based on perception data in the above method embodiment is implemented.

[0111] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of a local road information generation method based on perception data, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0112] The input device 630 may receive input digital or character information and generate signals related to user settings and function control of the image processing device. The output device 640 may include a display device such as a display screen.

[0113] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the method for generating local road information based on perception data in any of the above method embodiments is executed.

[0114] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0115] The electronic devices of the embodiments of the present invention exist in various forms, including but not limited to:

[0116] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0117] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0118] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0119] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0120] (5) Other electronic devices with data interaction functions.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating local road information based on perception data, characterized in that: include: Acquiring first data information, wherein the first data information includes sensed lane line data; Based on the first data information and the pre-trained lane information model, obtaining first lane information from the lane information model, where the first lane information includes lane information and lane topology relationship; A second lane information output is generated according to the first lane information, wherein the second lane information includes a lane boundary line.

2. The method according to claim 1, characterized in that The acquiring first lane information from the lane information model based on the first data information and the pre-trained lane information model includes: Preprocessing the first data information to obtain second data information, wherein the second data information includes the first data information in a specific expression form; The second data information is input into a pre-trained lane information model to obtain the first lane information from the lane information model.

3. The method according to claim 2, characterized in that The second data information includes a first grid expression corresponding to the perceived lane line data, wherein the first grid expression is represented by a grid image on which a polyline generated based on the perceived lane line data is drawn.

4. The method according to claim 2, characterized in that: The first data information also includes perceived road marking data, and the second data information includes perceived lane line data and a second grid expression corresponding to the perceived road marking data, and the second grid expression is represented by a grid image with a broken line generated based on the perceived lane line data and a marking generated based on the perceived road marking data.

5. The method according to claim 2, characterized in that: The second data information includes a first vector expression corresponding to the perceived lane line data, wherein the first vector expression is represented by a first vector set consisting of at least one vector generated based on the perceived lane line data.

6. The method according to claim 5, characterized in that The first data information also includes perceived road sign data, and the second data information also includes a second vector expression corresponding to the perceived road sign data, wherein the second vector expression is represented by a second vector set consisting of at least one vector generated based on the perceived road sign data.

7. The method according to any one of claims 1 to 6, characterized in that: The lane information model is trained by the following method: Acquire training data information, wherein the training data information includes first sample data information and true lane information, wherein the first sample data information includes sample-perceived lane line data, and the true lane information includes real lane information and real lane topology relationship extracted from a position covered by the sample-perceived lane line data in a high-precision map; Preprocessing the first sample data information to obtain second sample data information, wherein the second sample data information includes the first sample data information in a specific expression form; Inputting the second sample data information into a pre-selected machine learning model to obtain training lane information corresponding to the first sample data information from the machine learning model, and performing loss calculation based on the training lane information and the true lane information to obtain an adjustment coefficient of the machine learning model; The machine learning model is iteratively trained and adjusted according to the obtained adjustment coefficient to obtain the lane information model.

8. The method according to claim 7, characterized in that The first sample data information also includes sample-perceived road sign data.

9. The method according to claim 7, characterized in that: The training data information also includes auxiliary training data, and the auxiliary training data includes simulated perception lane data obtained by distorting lane lines extracted from positions covered by the sample perception lane line data in the high-precision map; The training method of the lane information model also includes: Inputting the auxiliary training data into a pre-selected machine learning model to obtain training lane information corresponding to the auxiliary training data from the machine learning model, and performing loss calculation based on the training lane information and true lane information to obtain an adjustment coefficient of the machine learning model; The machine learning model is iteratively trained and adjusted according to the obtained adjustment coefficient to obtain the lane information model.

10. The method according to any one of claims 1 to 6, characterized in that: The generating the second lane information according to the first lane information comprises: generating a virtual lane boundary according to the lane information in the first lane information; Bind the virtual lane boundary and the lane line detected by the perception module to generate the lane boundary line.

11. The method according to any one of claims 1 to 6, characterized in that: The second lane information also includes a lane topology verification result, and the generating the second lane information according to the first lane information also includes: The lane topology relationship in the first lane information is verified according to the lane information in the first lane information to obtain a lane topology relationship verification result.

12. A device for generating local road information based on perception data, characterized in that: include: A first data information acquisition module, used to acquire first data information, wherein the first data information includes sensed lane line data; A first lane information acquisition module, configured to acquire first lane information from a lane information model based on first data information and a pre-trained lane information model, wherein the first lane information includes lane information and a lane topology relationship; The second lane information generating module is used to generate second lane information output according to the first lane information, where the second lane information includes a lane boundary line.

13. A computing device, characterized in that include: A memory for storing executable instructions; as well as A processor, configured to execute executable instructions stored in a memory, wherein the executable instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. A mobile tool, characterized in that It includes the local road information generation device based on perception data as described in claim 12 or the computing device as described in claim 13.

15. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 11 are implemented.

16. A computer program product, comprising a computer program stored on a non-volatile computer-readable storage medium, characterized in that: The computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 11.

Images