Method for inferring lanes, method for training a lane inference model, and device

Abstract

This application relates to environmental perception technology in the field of autonomous driving in the field of artificial intelligence, and provides a method for inferring lanes, a method for training a lane inference model, and an apparatus. The method includes: obtaining detection results of a plurality of different types of sensors of a vehicle; performing coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system; using a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle. The method in the embodiments of this application helps to reduce the complexity of the neural network model used for lane inference.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and more particularly, to a method for inferring lanes, a method for training a lane inference model, and an apparatus therefor. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable machines to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0003] Autonomous driving is a mainstream application in the field of artificial intelligence. Autonomous driving technology relies on the collaborative cooperation of computer vision, radar, monitoring devices, and global positioning systems, etc., to enable motor vehicles to achieve autonomous driving without the need for active human operation. Autonomous vehicles use various computing systems to help transport passengers or goods from one location to another. Some autonomous vehicles may require some initial input or continuous input from an operator (such as a navigator, driver, or passenger). Autonomous vehicles allow the operator to switch from a manual operation mode to an autonomous driving mode or a mode in between. Since autonomous driving technology does not require humans to drive motor vehicles, it can theoretically effectively avoid human driving errors, reduce the occurrence of traffic accidents, and improve the transportation efficiency of roads. Therefore, autonomous driving technology has received increasing attention.

[0004] Whether a vehicle can obtain accurate and optional lanes available for autonomous driving is a key factor affecting the safety of autonomous driving. For example, training data can be collected in an offline environment to train a neural network model, and the trained neural network model can be used for lane inference. However, currently trained neural network models are often relatively complex. Summary of the Invention

[0005] The present application provides a method for inferring lanes, a method for training a lane inference model, and an apparatus therefor, which helps to reduce the complexity of the neural network model for lane inference.

[0006] In a first aspect, a method for inferring lanes is provided, and the method includes:

[0007] Obtain the detection results of multiple different types of sensors of the vehicle; perform coordinate system conversion on the detection results to obtain traffic element data in the same coordinate system; use a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.

[0008] In the prior art, when the neural network model needs to implement many functions, for example, the generalization ability for light conditions, road surface types, etc., and the processing of road surface recognition, the matching of road surface and driving intention, and dynamic and static obstacles all rely on the same neural network model for processing, which may lead to the neural network model being too complex.

[0009] For such a neural network model with a high degree of complexity, the design requirements of the model are also very high. At present, it is very difficult to design a neural network model that can reach the practical level and handle complex tasks (for example, a neural network model that can handle complex autonomous driving tasks).

[0010] In the embodiments of the present application, the traffic element data is data in the same coordinate system, which can avoid the neural network model being too complex due to the need to process multiple different types of data, thereby reducing the complexity of the neural network model used for lane inference.

[0011] It should be noted that the above-mentioned obtaining traffic element data detected by multiple different types of sensors may include: directly obtaining the original data detected by the sensors. That is, at this time, the detected original data is the traffic element data.

[0012] Alternatively, the above-mentioned obtaining traffic element data detected by multiple different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, the original data or original images obtained by the sensors), processing the detected detection results to obtain a processing result. At this time, the processing result obtained after processing is the traffic element data.

[0013] Combined with the first aspect, in some implementation manners of the first aspect, the obtaining traffic element data detected by multiple different types of sensors of the vehicle includes: obtaining the detection results of the multiple different types of sensors; processing the detection results to obtain the traffic element data.

[0014] In the embodiments of the present application, obtaining the detection results of the multiple different types of sensors and processing the detection results to obtain the traffic element data in the same coordinate system can avoid the neural network model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby reducing the complexity of the neural network model used for lane inference.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the processing of the detection result to obtain the traffic element data includes: performing coordinate system conversion on the detection result to obtain the traffic element data.

[0016] In the embodiments of the present application, performing coordinate system conversion on the detection result can convert the detection result to the same coordinate system, which can avoid the neural network model being too complex due to the need to directly process the detection result (the detection result includes various different types of data), thereby reducing the complexity of the neural network model for lane inference.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the various different types of sensors include cameras, lidars, or millimeter-wave radars.

[0018] In combination with the first aspect, in certain implementations of the first aspect, the traffic element data is the outline, position, and / or movement trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles, pedestrians, roads, or traffic lights around the vehicle.

[0019] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: determining an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a panoramic view image or a front view image; wherein, using the preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using the preset neural network model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

[0020] In the embodiments of the present application, the environment around the vehicle can be intuitively represented by the environmental perception image. At this time, performing lane inference based on the environmental perception image can make the inferred lane (of the vehicle) more accurate.

[0021] Optionally, the determining the environmental perception image according to the traffic element data includes: based on the traffic element data, superimposing the traffic elements onto an initial image (for example, a panoramic view image obtained by stitching one or more frames of perception images collected by a sensor, or an initial environmental image generated with the vehicle as the center and the environment blank) to obtain the environmental perception image.

[0022] In combination with the first aspect, in some implementations of the first aspect, using a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel on.

[0023] In the embodiments of the present application, the driving intention of the vehicle is used to indicate the route that the vehicle is to travel on. Performing lane inference based on the traffic element data and the driving intention of the vehicle can further improve the accuracy of the inferred (vehicle's) lane.

[0024] In combination with the first aspect, in some implementations of the first aspect, the driving intention of the vehicle is a schematic diagram of the route of the lane that the vehicle is to travel on or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0025] In the embodiments of the present application, the driving intention of the vehicle is a schematic diagram of the route or a vehicle semantic indication. Through the driving intention of the vehicle, the route that the vehicle is to travel on can be intuitively and accurately indicated.

[0026] In combination with the first aspect, in some implementations of the first aspect, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0027] In the embodiments of the present application, by representing with the center line of the lane and the width of the lane or a road surface cost map, the lane of the vehicle can be intuitively and accurately represented.

[0028] In a second aspect, a method for training a lane inference model is provided. The method includes:

[0029] Obtaining the detection results of various different types of sensors of a vehicle; performing coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system; using a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle; and adjusting the parameters of the lane inference model according to the lane of the vehicle.

[0030] In the prior art, when a neural network model needs to implement many functions, for example, the generalization capabilities for light conditions, road surface types, etc., as well as road surface recognition, the matching of the road surface and the driving intention, and the handling of dynamic and static obstacles all rely on the same neural network model for processing, which may lead to the neural network model being too complex.

[0031] For such a neural network model with high complexity, the design requirements for the model are also very high. Currently, it is very difficult to design a neural network model that can reach the practical level and handle complex tasks (for example, a neural network model that can handle complex autonomous driving tasks).

[0032] In the embodiments of the present application, the traffic element data is data in the same coordinate system, which can avoid the lane inference model being too complex due to the need to process multiple different types of data, thereby reducing the complexity of the lane inference model.

[0033] It should be noted that the above-mentioned obtaining traffic element data detected by multiple different types of sensors may include: directly obtaining the original data detected by the sensors. That is to say, at this time, the detected original data is the traffic element data.

[0034] Or, the above-mentioned obtaining traffic element data detected by multiple different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, the original data or original images obtained by the sensors), processing the detected detection results to obtain a processing result. At this time, the processing result obtained after processing is the traffic element data.

[0035] Combined with the second aspect, in some implementation manners of the second aspect, the obtaining traffic element data detected by multiple different types of sensors of the vehicle includes: obtaining the detection results of the multiple different types of sensors; processing the detection results to obtain the traffic element data.

[0036] In the embodiments of the present application, obtaining the detection results of the multiple different types of sensors and processing the detection results to obtain the traffic element data in the same coordinate system can avoid the lane inference model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby reducing the complexity of the lane inference model.

[0037] Combined with the second aspect, in some implementation manners of the second aspect, the processing the detection results to obtain the traffic element data includes: performing coordinate system conversion on the detection results to obtain the traffic element data.

[0038] In the embodiments of the present application, performing coordinate system conversion on the detection results can convert the detection results to the same coordinate system, which can avoid the lane inference model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby reducing the complexity of the lane inference model.

[0039] In combination with the second aspect, in some implementations of the second aspect, the multiple different types of sensors include cameras, lidar, or millimeter-wave radars.

[0040] In combination with the second aspect, in some implementations of the second aspect, the traffic element data is the outline, position, and / or movement trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads, or traffic lights.

[0041] In combination with the second aspect, in some implementations of the second aspect, the method further includes: determining an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a surround view image or a front view image; wherein, the using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using the lane inference model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

[0042] In the embodiments of the present application, the environment around the vehicle can be intuitively represented by the environmental perception image. At this time, performing lane inference based on the environmental perception image can make the inferred lane (of the vehicle) more accurate.

[0043] Optionally, the determining an environmental perception image according to the traffic element data includes: based on the traffic element data, superimposing the traffic elements onto an initial image (for example, a surround view image obtained by stitching one or more frames of perception images collected by a sensor, or an initial environmental image generated with the vehicle as the center and the environment blank) to obtain the environmental perception image.

[0044] In combination with the second aspect, in some implementations of the second aspect, the using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

[0045] In the embodiments of the present application, the driving intention of the vehicle is used to indicate the route that the vehicle is to travel. Performing lane inference based on the traffic element data and the driving intention of the vehicle can further improve the accuracy of the inferred lane (of the vehicle).

[0046] In combination with the second aspect, in some implementation manners of the second aspect, the driving intention of the vehicle is a route schematic diagram of the lane to be traveled by the vehicle or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0047] In the embodiments of the present application, the driving intention of the vehicle is a route schematic diagram or a vehicle semantic indication, and through the driving intention of the vehicle, the route to be traveled by the vehicle can be intuitively and accurately indicated.

[0048] In combination with the second aspect, in some implementation manners of the second aspect, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0049] In the embodiments of the present application, by representing with the center line of the lane and the width of the lane or a road surface cost map, the lane of the vehicle can be intuitively and accurately represented.

[0050] In a third aspect, a device for inferring a lane is provided, including:

[0051] An acquisition unit, configured to acquire detection results of a plurality of different types of sensors of a vehicle; perform coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system; a lane inference unit, configured to use a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.

[0052] In the embodiments of the present application, the traffic element data is data in the same coordinate system, which can avoid the neural network model being too complex due to the need to process a variety of different types of data, thereby being able to reduce the complexity of the neural network model for lane inference.

[0053] It should be noted that the above acquisition of traffic element data detected by a plurality of different types of sensors may include: directly acquiring the original data detected by the sensors, and at this time, the detected original data is the traffic element data.

[0054] Or, the above acquisition of traffic element data detected by a plurality of different types of sensors may also include: after acquiring the detection results detected by the sensors (for example, the original data or original image acquired by the sensors, etc.), performing processing on the detected detection results to obtain a processing result, and at this time, the processing result obtained after processing is the traffic element data.

[0055] In combination with the third aspect, in some implementation manners of the third aspect, the obtaining unit is specifically configured to: obtain the detection results of the multiple different types of sensors; and process the detection results to obtain the traffic element data.

[0056] In the embodiment of the present application, obtaining the detection results of the multiple different types of sensors and processing the detection results to obtain the traffic element data in the same coordinate system can avoid the neural network model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby being able to reduce the complexity of the neural network model for lane inference.

[0057] In combination with the third aspect, in some implementation manners of the third aspect, the obtaining unit is specifically configured to: perform coordinate system conversion on the detection results to obtain the traffic element data.

[0058] In the embodiment of the present application, performing coordinate system conversion on the detection results can convert the detection results to the same coordinate system, which can avoid the neural network model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby being able to reduce the complexity of the neural network model for lane inference.

[0059] In combination with the third aspect, in some implementation manners of the third aspect, the multiple different types of sensors include a camera, a lidar, or a millimeter-wave radar.

[0060] In combination with the third aspect, in some implementation manners of the third aspect, the traffic element data is the contour, position, and / or movement trajectory of a traffic element, and the traffic element includes one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads, or traffic lights.

[0061] In combination with the third aspect, in some implementation manners of the third aspect, the device further includes a superimposing unit, configured to: determine an environment perception image according to the traffic element data, where the environment perception image is used to indicate the environment around the vehicle and the traffic elements, and the environment perception image includes at least one of a panoramic image or a front view image; wherein, the lane inference unit is specifically configured to: use a preset neural network model to perform lane inference based on the environment perception image to obtain the lane of the vehicle.

[0062] In the embodiment of the present application, the environment around the vehicle can be intuitively represented by the environment perception image. At this time, performing lane inference based on the environment perception image can make the inferred lane (of the vehicle) more accurate.

[0063] In combination with the third aspect, in some implementation manners of the third aspect, the lane inference unit is specifically configured to: use a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle, so as to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel along.

[0064] In the embodiments of the present application, the driving intention of the vehicle is used to indicate the route that the vehicle is to travel along. Performing lane inference based on the traffic element data and the driving intention of the vehicle can further improve the accuracy of the inferred lane (of the vehicle).

[0065] In combination with the third aspect, in some implementation manners of the third aspect, the driving intention of the vehicle is a route schematic diagram of the lane that the vehicle is to travel along or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0066] In the embodiments of the present application, the driving intention of the vehicle is a route schematic diagram or a vehicle semantic indication. Through the driving intention of the vehicle, the route that the vehicle is to travel along can be intuitively and accurately indicated.

[0067] In combination with the third aspect, in some implementation manners of the third aspect, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0068] In the embodiments of the present application, by representing with the center line of the lane and the width of the lane or a road surface cost map, the lane of the vehicle can be intuitively and accurately represented.

[0069] Fourth aspect, there is provided an apparatus for training a lane inference model, including:

[0070] An acquisition unit, configured to acquire detection results of various different types of sensors of a vehicle; perform coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system; a lane inference unit, configured to use a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle; and an adjustment unit, configured to adjust parameters of the lane inference model according to the lane of the vehicle.

[0071] In the embodiments of the present application, the traffic element data is data in the same coordinate system, which can avoid the lane inference model from being too complex due to the need to process various different types of data, thereby being able to reduce the complexity of the lane inference model.

[0072] It should be noted that the traffic element data obtained from the above-mentioned multiple different types of sensors may include: directly obtaining the original data detected by the sensors. That is to say, at this time, the detected original data is the traffic element data.

[0073] Alternatively, the traffic element data obtained from the above-mentioned multiple different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, the original data or original images obtained by the sensors), processing the detected detection results to obtain a processing result. At this time, the processing result obtained after processing is the traffic element data.

[0074] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the obtaining unit is specifically configured to: obtain the detection results of the multiple different types of sensors; process the detection results to obtain the traffic element data.

[0075] In the embodiments of the present application, obtaining the detection results of the multiple different types of sensors and processing the detection results to obtain the traffic element data in the same coordinate system can avoid the lane inference model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby being able to reduce the complexity of the lane inference model.

[0076] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the obtaining unit is specifically configured to: perform coordinate system conversion on the detection results to obtain the traffic element data.

[0077] In the embodiments of the present application, performing coordinate system conversion on the detection results can convert the detection results to the same coordinate system, which can avoid the lane inference model being too complex due to the need to directly process the detection results (the detection results include multiple different types of data), thereby being able to reduce the complexity of the lane inference model.

[0078] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the multiple different types of sensors include cameras, lidars or millimeter wave radars.

[0079] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the traffic element data is the contour, position and / or movement trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads or traffic lights.

[0080] In combination with the fourth aspect, in some implementations of the fourth aspect, the device further includes a superimposing unit, configured to: determine an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a surround view image or a front view image; wherein, the lane inference unit is specifically configured to: use the lane inference model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

[0081] In the embodiments of the present application, the environment around the vehicle can be intuitively represented by the environmental perception image. At this time, performing lane inference based on the environmental perception image can make the inferred lane (of the vehicle) more accurate.

[0082] In combination with the fourth aspect, in some implementations of the fourth aspect, the lane inference unit is specifically configured to: use the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

[0083] In the embodiments of the present application, the driving intention of the vehicle is used to indicate the route that the vehicle is to travel. Performing lane inference based on the traffic element data and the driving intention of the vehicle can further improve the accuracy of the inferred lane (of the vehicle).

[0084] In combination with the fourth aspect, in some implementations of the fourth aspect, the driving intention of the vehicle is a route schematic diagram of the lane that the vehicle is to travel or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0085] In the embodiments of the present application, the driving intention of the vehicle is a route schematic diagram or a vehicle semantic indication, and the route that the vehicle is to travel can be intuitively and accurately indicated through the driving intention of the vehicle.

[0086] In combination with the fourth aspect, in some implementations of the fourth aspect, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0087] In the embodiments of the present application, it can be intuitively and accurately represented by the center line of the lane and the width of the lane or the road surface cost map.

[0088] In a fifth aspect, a device for inferring lanes is provided. The device includes a storage medium and a central processing unit. The storage medium can be a non-volatile storage medium. A computer-executable program is stored in the storage medium. The central processing unit is connected to the non-volatile storage medium and executes the computer-executable program to implement the method in any possible implementation manner of the first aspect.

[0089] In a sixth aspect, a device for training a lane inference model is provided. The device includes a storage medium and a central processing unit. The storage medium can be a non-volatile storage medium. A computer-executable program is stored in the storage medium. The central processing unit is connected to the non-volatile storage medium and executes the computer-executable program to implement the method in any possible implementation manner of the second aspect.

[0090] In a seventh aspect, a chip is provided. The chip includes a processor and a data interface. The processor reads instructions stored on a memory through the data interface and executes the method in any possible implementation manner of the first aspect or the second aspect.

[0091] Optionally, as an implementation manner, the chip may further include a memory. Instructions are stored in the memory. The processor is configured to execute the instructions stored on the memory. When the instructions are executed, the processor is configured to execute the method in any possible implementation manner of the first aspect or the second aspect.

[0092] In an eighth aspect, a computer-readable storage medium is provided. The computer-readable medium stores program code for a device to execute. The program code includes instructions for executing the method in any possible implementation manner of the first aspect or the second aspect.

[0093] In a ninth aspect, a vehicle is provided. The vehicle includes any possible device in the third aspect or the fourth aspect described above.

[0094] In the embodiments of the present application, the traffic element data is data in the same coordinate system, which can avoid the neural network model from being too complex due to the need to process various different types of data, thereby being able to reduce the complexity of the neural network model for lane inference. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is a schematic structural diagram of an autonomous vehicle provided by an embodiment of the present application;

[0096] Figure 2 is a schematic structural diagram of an autonomous driving system provided by an embodiment of the present application;

[0097] Figure 3Schematic diagram of the structure of a neural network processor provided by an embodiment of the present application;

[0098] Figure 4 Schematic diagram of the application of a cloud-side instruction autonomous driving vehicle provided by an embodiment of the present application;

[0099] Figure 5 Schematic diagram of the structure of an autonomous driving system provided by an embodiment of the present application;

[0100] Figure 6 Schematic block diagram of a road structure recognition module provided by an embodiment of the present application;

[0101] Figure 7 Schematic block diagram of a method for inferring lanes provided by an embodiment of the present application;

[0102] Figure 8 Schematic flowchart of a method for inferring lanes provided by another embodiment of the present application;

[0103] Figure 9 Schematic flowchart of processing the detection results of sensors in an embodiment of the present application;

[0104] Figure 10 Schematic flowchart of determining an environmental perception image in an embodiment of the present application;

[0105] Figure 11 Schematic flowchart of determining an environmental perception image in another embodiment of the present application;

[0106] Figure 12 Schematic flowchart of determining the driving intention of a vehicle in an embodiment of the present application;

[0107] Figure 13 Schematic flowchart of determining the driving intention of a vehicle in another embodiment of the present application;

[0108] Figure 14 Schematic block diagram of a route schematic diagram generated in an embodiment of the present application;

[0109] Figure 15 Schematic block diagram of using a neural network model for lane inference in an embodiment of the present application;

[0110] Figure 16 Schematic block diagram of using a neural network model for lane inference in another embodiment of the present application;

[0111] Figure 17 Schematic block diagram of a method for training a lane inference model provided by an embodiment of the present application;

[0112] Figure 18 It is a schematic block diagram of a device for inferring lanes provided by an embodiment of the present application;

[0113] Figure 19 It is a schematic block diagram of a device for training a driving behavior decision-making model provided by an embodiment of the present application;

[0114] Figure 20 It is a schematic block diagram of a device for an autonomous vehicle provided by an embodiment of the present application. Specific embodiments

[0115] Next, the technical solutions in the present application will be described in conjunction with the accompanying drawings.

[0116] The technical solutions of the embodiments of the present application can be applied to various vehicles. Specifically, the vehicle can be an internal combustion locomotive, a smart electric vehicle, or a hybrid vehicle. Alternatively, the vehicle can also be a vehicle of other power types, etc. The embodiments of the present application do not limit this.

[0117] The vehicle in the embodiments of the present application can be an autonomous vehicle. For example, the autonomous vehicle can be configured with an autonomous driving mode, which can be a full autonomous driving mode or a partial autonomous driving mode. The embodiments of the present application do not limit this.

[0118] The vehicle in the embodiments of the present application can also be configured with other driving modes. The other driving modes can include one or more of a sports mode, an economy mode, a standard mode, a snow mode, and a climbing mode, etc. The autonomous vehicle can switch between the autonomous driving mode and the above-mentioned multiple (driving models for the driver to drive the vehicle). The embodiments of the present application do not limit this.

[0119] Figure 1 It is a functional block diagram of vehicle 100 provided by an embodiment of the present application.

[0120] In one embodiment, vehicle 100 is configured to be in a full or partial autonomous driving mode.

[0121] For example, vehicle 100 can control itself while in the autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through manual operation, determine the possible behaviors of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the possibility of the other vehicle performing the possible behavior, and control vehicle 100 based on the determined information. When vehicle 100 is in the autonomous driving mode, vehicle 100 can be set to operate without interacting with people.

[0122] Vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, and a power source 110, a computer system 112, and a user interface 116.

[0123] Optionally, vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. Additionally, each subsystem and component of vehicle 100 may be interconnected by wire or wirelessly.

[0124] Propulsion system 102 may include components that provide powered movement for vehicle 100. In one embodiment, propulsion system 102 may include an engine 118, an energy source 119, a transmission 120, and wheels / tires 121. Engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine composed of a gasoline engine and an electric motor, or a hybrid engine composed of an internal combustion engine and an air compression engine. Engine 118 converts energy source 119 into mechanical energy.

[0125] Examples of energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other power sources. Energy source 119 may also provide energy for other systems of vehicle 100.

[0126] Transmission 120 may transfer mechanical power from engine 118 to wheels 121. Transmission 120 may include a gearbox, a differential, and a drive shaft.

[0127] In one embodiment, transmission 120 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

[0128] Sensor system 104 may include several sensors that sense information about the environment around vehicle 100.

[0129] For example, sensor system 104 may include a positioning system 122 (the positioning system may be a GPS system, a Beidou system, or other positioning systems), an inertial measurement unit (IMU) 124, a radar 126, a lidar 128, and a camera 130. Sensor system 104 may also include sensors that monitor the internal systems of vehicle 100 (such as an in-vehicle air quality monitor, a fuel gauge, an engine oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.). Such detection and identification are key functions for the safe operation of autonomous vehicle 100.

[0130] The positioning system 122 can be used to estimate the geographical location of the vehicle 100. The IMU 124 is used to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope.

[0131] The radar 126 can utilize radio signals to sense objects within the surrounding environment of the vehicle 100. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or forward direction of the objects.

[0132] The lidar 128 can utilize lasers to sense objects in the environment where the vehicle 100 is located. In some embodiments, the lidar 128 can include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components.

[0133] The camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a static camera or a video camera.

[0134] The control system 106 is for controlling the operation of the vehicle 100 and its components. The control system 106 can include various elements, including a steering system 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a route control system 142, and an obstacle avoidance system 144.

[0135] The steering system 132 is operable to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system.

[0136] The throttle 134 is used to control the operating speed of the engine 118 and thus control the speed of the vehicle 100.

[0137] The braking unit 136 is used to control the deceleration of the vehicle 100. The braking unit 136 can use friction to slow down the wheels 121. In other embodiments, the braking unit 136 can convert the kinetic energy of the wheels 121 into electric current. The braking unit 136 can also take other forms to slow down the rotational speed of the wheels 121 so as to control the speed of the vehicle 100.

[0138] The computer vision system 140 can be operated to process and analyze the images captured by the camera 130 in order to identify objects and / or features in the surrounding environment of the vehicle 100. The objects and / or features can include traffic signals, road boundaries, and obstacles. The computer vision system 140 can use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 can be used to map the environment, track objects, estimate the speed of objects, and so on.

[0139] The route control system 142 is used to determine the driving route of the vehicle 100. In some embodiments, the route control system 142 may combine data from the sensors 138, GPS 122, and one or more pre - determined maps to determine the driving route for the vehicle 100.

[0140] The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise cross potential obstacles in the environment of the vehicle 100.

[0141] Of course, in one example, the control system 106 may additionally or alternatively include components other than those shown and described. Or some of the above - shown components may be reduced.

[0142] The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripheral device 108. The peripheral device 108 may include a wireless communication system 146, an on - vehicle computer 148, a microphone 150, and / or a speaker 152.

[0143] In some embodiments, the peripheral device 108 provides a means for the user of the vehicle 100 to interact with the user interface 116. For example, the on - vehicle computer 148 may provide information to the user of the vehicle 100. The user interface 116 may also operate the on - vehicle computer 148 to receive user input. The on - vehicle computer 148 may be operated through a touch screen. In other cases, the peripheral device 108 may provide a means for the vehicle 100 to communicate with other devices located inside the vehicle. For example, the microphone 150 may receive audio (e.g., voice commands or other audio inputs) from the user of the vehicle 100. Similarly, the speaker 152 may output audio to the user of the vehicle 100.

[0144] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as CDMA, EVD0, GSM / GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system 146 may utilize WiFi to communicate with a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 may communicate directly with devices using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0145] Power source 110 can supply power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such a battery can be configured to supply power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, such as in some all-electric vehicles.

[0146] Some or all functions of vehicle 100 are controlled by computer system 112. Computer system 112 can include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium such as data storage device 114. Computer system 112 can also be a plurality of computing devices that control individual components or subsystems of vehicle 100 in a distributed manner.

[0147] Processor 113 can be any conventional processor, such as a commercially available CPU. Alternatively, the processor can be a special-purpose device such as an ASIC or other hardware-based processor. Although Figure 1 the functional diagram shows the processor, memory, and other elements of computer 110 in the same block, those of ordinary skill in the art should understand that the processor, computer, or memory can actually include multiple processors, computers, or memories within the same physical housing. For example, the memory can be a hard disk drive or other storage medium located in a housing different from that of computer 110. Thus, a reference to a processor or computer will be understood to include a reference to a collection of processors or computers or memories that may or may not operate in parallel. Instead of using a single processor to perform the steps described herein, some components such as the steering component and the deceleration component can each have their own processor that only performs calculations related to component-specific functions.

[0148] In various aspects described herein, the processor can be located remote from the vehicle and communicate wirelessly with the vehicle. In other aspects, some of the processes described herein are executed on a processor disposed within the vehicle while others are executed by a remote processor, including taking the necessary steps to perform a single maneuver.

[0149] In some embodiments, data storage device 114 can contain instructions 115 (e.g., program logic) that can be executed by processor 113 to perform various functions of vehicle 100, including those functions described above. Data storage device 114 can also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of propulsion system 102, sensor system 104, control system 106, and peripheral device 108.

[0150] In addition to instruction 115, the data storage device 114 may also store data, such as road maps, route information, the position, direction, speed of the vehicle, and other such vehicle data, as well as other information. Such information may be used by the vehicle 100 and the computer system 112 during operation of the vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0151] A user interface 116 for providing information to or receiving information from a user of the vehicle 100. Optionally, the user interface 116 may include one or more input / output devices within the set of peripheral devices 108, such as a wireless communication system 146, a vehicle-to-vehicle computer 148, a microphone 150, and a speaker 152.

[0152] The computer system 112 may control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, and the control system 106) and from the user interface 116. For example, the computer system 112 may utilize inputs from the control system 106 to control the steering unit 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, the computer system 112 may operate to provide control over many aspects of the vehicle 100 and its subsystems.

[0153] Optionally, one or more of the above components may be separately installed or associated with the vehicle 100. For example, the data storage device 114 may exist partially or completely separately from the vehicle 100. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0154] Optionally, the above components are just an example. In actual applications, the components in each of the above modules may be added or deleted according to actual needs. Figure 1 It should not be construed as a limitation to the embodiments of the present application.

[0155] An autonomous vehicle traveling on a road, such as the vehicle 100 above, can identify objects within its surrounding environment to determine adjustments to its current speed. The objects may be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object may be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., can be used to determine the speed adjustment to be made by the autonomous vehicle.

[0156] Optionally, the autonomous vehicle 100 or a computing device associated with the autonomous vehicle 100 (such as Figure 1A computer system 112, a computer vision system 140, and a data storage device 114) can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of the others, so all the identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified object. In other words, an autonomous vehicle can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, etc.

[0157] In addition to providing instructions to adjust the speed of the autonomous vehicle, the computing device can also provide instructions to modify the steering angle of the vehicle 100 so that the autonomous vehicle follows a given trajectory and / or maintains a safe lateral and longitudinal distance from the objects near the autonomous vehicle (e.g., a sedan in an adjacent lane on the road).

[0158] The above vehicle 100 can be a sedan, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, a recreational vehicle, a playground vehicle, construction equipment, a tram, a golf cart, a train, and a trolley, etc., and the embodiments of the present application do not make special limitations.

[0159] Figure 2 is a schematic diagram of the autonomous driving system provided by the embodiments of the present application.

[0160] As Figure 2The autonomous driving system shown includes a computer system 101. Among them, the computer system 101 includes a processor 103, and the processor 103 is coupled to a system bus 105. The processor 103 can be one or more processors, and each processor can include one or more processor cores. A display adapter 107, which can drive a display 109, and the display 109 is coupled to the system bus 105. The system bus 105 is coupled to an input / output (I / O) bus 113 through a bus bridge 111. An I / O interface 115 is coupled to the I / O bus. The I / O interface 115 communicates with a variety of I / O devices, such as an input device 117 (e.g., keyboard, mouse, touch screen, etc.), a media tray 121 (e.g., CD-ROM, multimedia interface, etc.), a transceiver 123 (which can send and / or receive radio communication signals), a camera 155 (which can capture static and dynamic digital video images), and an external USB interface 125. Optionally, the interface connected to the I / O interface 115 can be a USB interface.

[0161] Among them, the processor 103 can be any conventional processor, including a reduced instruction set computer (RISC) processor, a complex instruction set computer (CISC) processor, or a combination of the above. Optionally, the processor can be a dedicated device such as an application specific integrated circuit (ASIC). Optionally, the processor 103 can be a neural network processor or a combination of a neural network processor and the above conventional processors.

[0162] Optionally, in various embodiments described herein, the computer system 101 can be located away from the autonomous vehicle and can communicate wirelessly with the autonomous vehicle. In other aspects, some of the processes described herein are executed on a processor provided in the autonomous vehicle, and others are executed by a remote processor, including taking actions required to perform a single maneuver.

[0163] The computer 101 can communicate with a software deployment server 149 through a network interface 129. The network interface 129 is a hardware network interface, such as a network card. The network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, the network 127 can also be a wireless network, such as a WiFi network, a cellular network, etc.

[0164] The hard disk drive interface is coupled to the system bus 105. The hard disk drive interface is connected to the hard disk drive. The system memory 135 is coupled to the system bus 105. The data running in the system memory 135 may include the operating system 137 and application programs 143 of the computer 101.

[0165] The operating system includes a parser 139 (shell) and a kernel 141. The shell is an interface between the user and the kernel of the operating system. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waits for the user's input, interprets the user's input to the operating system, and processes various output results of the operating system.

[0166] The kernel 141 consists of those parts in the operating system that are used to manage memory, files, peripherals, and system resources. Interacting directly with the hardware, the operating system kernel usually runs processes, provides inter-process communication, provides CPU time slice management, interrupts, memory management, IO management, etc.

[0167] The application program 143 includes programs related to controlling the automatic driving of a vehicle. For example, programs for managing the interaction between an automatically driven vehicle and road obstacles, programs for controlling the route or speed of an automatically driven vehicle, and programs for controlling the interaction between an automatically driven vehicle and other automatically driven vehicles on the road. The application program 143 also exists on the system of the deploying server 149. In one embodiment, when the application program 143 needs to be executed, the computer system 101 can download the application program 143 from the deploying server 149.

[0168] The sensor 153 is associated with the computer system 101. The sensor 153 is used to detect the environment around the computer 101. For example, the sensor 153 can detect animals, vehicles, obstacles, crosswalks, etc. Further, the sensor can also detect the environment around the above-mentioned animals, vehicles, obstacles, crosswalks and other objects. For example, the environment around an animal, such as other animals appearing around the animal, weather conditions, the brightness of the surrounding environment, etc. Optionally, if the computer 101 is located on an autonomous vehicle, the sensor can be a camera, an infrared sensor, a chemical detector, a microphone, etc.

[0169] For example, the application program 143 can perform lane inference based on the surrounding environment information detected by the sensor 153 to obtain the lane to be traveled (or the lane that can be traveled) by the vehicle. At this time, based on the lane to be traveled, path planning can be performed on the autonomous vehicle, and then the control amount of the vehicle can be determined to achieve the automatic driving of the vehicle.

[0170] Figure 3It is a hardware structure diagram of a chip provided by an embodiment of the present application. The chip includes a neural network processor 20. The chip can be, for example, Figure 2 in the processor 103 shown, and is used to perform lane inference according to environmental information. In the embodiments of the present application, the algorithms of each layer in the pre-trained neural network can be implemented in the chip shown in Figure 3 .

[0171] The method for inferring lanes in the embodiments of the present application can also be implemented in the chip shown in Figure 3 . Among them, the chip can be the same chip as the chip that implements the above-mentioned pre-trained neural network, or the chip can also be a different chip from the chip that implements the above-mentioned pre-trained neural network. The embodiments of the present application do not limit this.

[0172] The neural network processor NPU 50 is mounted on the host CPU as a coprocessor, and tasks are allocated by the Host CPU. The core part of the NPU is the arithmetic circuit 50, and the controller 504 controls the arithmetic circuit 503 to extract matrix data from the memory and perform multiplication operations.

[0173] In some implementations, the arithmetic circuit 203 internally includes multiple processing units (process engine, PE). In some implementations, the arithmetic circuit 203 is a two-dimensional systolic array. The arithmetic circuit 203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 203 is a general matrix processor.

[0174] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 202 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 201 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are stored in the accumulator 208.

[0175] The vector calculation unit 207 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, etc. For example, the vector calculation unit 207 can be used for network calculations in non-convolutional / non-FC layers of the neural network, such as pooling, batch normalization, local response normalization, etc.

[0176] In some implementations, the vector computing unit 207 can store the processed output vectors in the unified buffer 206. For example, the vector computing unit 207 can apply a non-linear function to the output of the arithmetic circuit 203, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computing unit 207 generates normalized values, combined values, or both. In some implementations, the processed output vectors can be used as activation inputs to the arithmetic circuit 203, for example, for use in subsequent layers of a neural network.

[0177] The unified memory 206 is used to store input data and output data.

[0178] The weight data directly transfers the input data in the external memory to the input memory 201 and / or the unified memory 206, stores the weight data in the external memory into the weight memory 202, and stores the data in the unified memory 206 into the external memory through the memory access controller 205 (direct memory access controller, DMAC).

[0179] The bus interface unit (BIU) 210 is used to interact between the main CPU, DMAC, and the instruction fetch memory 209 through the bus.

[0180] The instruction fetch buffer 209 connected to the controller 204 is used to store the instructions used by the controller 204;

[0181] The controller 204 is used to call the instructions cached in the instruction fetch memory 209 to control the working process of the arithmetic accelerator.

[0182] Generally, the unified memory 206, the input memory 201, the weight memory 202, and the instruction fetch memory 209 are all on-chip memories, and the external memory is the memory outside the NPU. The external memory can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM), or other readable and writable memories.

[0183] The computer system 112 can also receive information from other computer systems or transfer information to other computer systems. Alternatively, the sensor data collected from the sensor system 104 of the vehicle 100 can be transferred to another computer for processing this data.

[0184] For example, as Figure 4 shown, data from the computer system 312 can be transmitted via a network to the server 320 on the cloud side for further processing. The network and intermediate nodes can include various configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using proprietary communication protocols of one or more companies, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing. Such communication can be performed by any device capable of transmitting data to and from other computers, such as modems and wireless interfaces.

[0185] In one example, the server 320 can include a server having multiple computers, for example, a load-balanced server cluster, which exchanges information with different nodes of the network for the purpose of receiving, processing, and transmitting data from the computer system 312. The server can be configured similarly to the computer system 312, having a processor 330, a memory 340, instructions 350, and data 360.

[0186] Exemplarily, the data 360 of the server 320 can include map information and / or information related to the road conditions around the autonomous vehicle. For example, the server 320 can receive, detect, store, update, and transmit information related to the road conditions of the autonomous vehicle.

[0187] For example, the map information can be a high-precision map, and the map information can include pre-stored lane line information (e.g., lane identification lines).

[0188] Again, for example, the information related to the road conditions around the autonomous vehicle includes the position information and motion parameter information of other vehicles having intersections with the predicted driving path of the autonomous vehicle. For example, the current position information of other vehicles, the speed information of other vehicles, the predicted driving trajectory information of other vehicles, etc.

[0189] Figure 5 This is an exemplary structural block diagram of an autonomous driving system 500 according to an embodiment of the present application. As Figure 5 shown, the autonomous driving system 500 can include a sensor 510, a perception positioning map subsystem 520, a fusion subsystem 530, a prediction and cognition subsystem 540, a planning subsystem 550, a control subsystem 560, and an actuator 570.

[0190] It should be noted that Figure 5 the shown autonomous driving system 500 is only an example and not a limitation. Those of ordinary skill in the art can understand that the autonomous driving system 500 can include more or fewer modules or subsystems (or units), and the embodiments of the present application do not limit this.

[0191] The sensor 510 can be used to sense objects outside a vehicle (e.g., an autonomous vehicle) and transmit the sensed data to the perception and positioning map module 520. Among them, the sensor 510 can include sensors such as cameras, millimeter-wave radars, lidars, and ultrasonic radars.

[0192] The perception and positioning map subsystem 520 can be used to obtain traffic element data in the environment, provide the positioning of the vehicle (e.g., an autonomous vehicle), and the road topology information where the vehicle is located.

[0193] As Figure 5 shown, the perception and positioning map subsystem 520 includes a visual perception module 522, a radar perception module 524, a positioning module 526, and a map module 526. Among them, the visual perception module 522 can be used to detect lane marking lines, moving vehicles, passable areas, and traffic signs. The radar perception module 524 can be used to detect vehicles, road edges, and passable areas. The positioning module 526 can be used to achieve the global positioning of the vehicle. The map module 526 can be used to obtain the surrounding road topology.

[0194] The fusion subsystem 530 can be used to fuse the output results of multiple perception modules to achieve more accurate detection of moving targets, detection of passable areas, and detection of road edges.

[0195] As Figure 5 shown, the fusion subsystem 530 includes a target fusion module 532, a passable road surface fusion module 534, and a road edge detection and fusion module 536. Among them, the target fusion module 532 can be used to achieve the fusion and tracking of visual perception targets and radar detection targets. The passable road surface fusion module 534 can be used to achieve the fusion and tracking of visually detected passable areas and radar-detected passable areas. The road edge detection and fusion module 536 can be used to fuse and track the visual semantic segmentation results and the radar-detected road edges.

[0196] The prediction and cognition subsystem 540 can be used to achieve further understanding and reasoning of the environment.

[0197] As Figure 5 shown, the prediction and cognition subsystem 540 includes a target prediction module 542 and a road structure cognition module 544. Among them, the target prediction module 542 can be used to achieve the future motion prediction of detected moving targets. The road structure cognition module 544 can be used to perform lane reasoning to obtain the lane to be traveled (or the lane that can be traveled) by the autonomous vehicle.

[0198] The planning subsystem 550 can be used to plan the driving strategy of the host vehicle based on the obtained environmental detection results and environmental cognition results.

[0199] As Figure 5As shown in the figure, the planning subsystem 550 includes a highly automated driving (HAD) planner 552, a partial automated driving (PAD) planner 554, and a function manager 556. Among them, the HAD planner 552 can be used for planning in the case of highly automated driving functions, which includes more automated driving functions. The PAD planner 554 can be used for partial automated driving, such as functions like adaptive cruise control. The function manager 556 can be used to start and switch between HAD functions and PAD functions.

[0200] The control subsystem 560 is used to calculate the vehicle body control amount based on the planning result, and thus send a target control amount instruction to the chassis electronic control system.

[0201] Such as Figure 5 As shown in the figure, the control subsystem 560 includes a lateral control module 562, a longitudinal control module 564, and an arbiter 566. Among them, the lateral control module 562 can be used to calculate the steering system control amount for achieving the target trajectory. The longitudinal control module 564 can be used to calculate the power and braking system control amounts for achieving the target speed curve. The arbiter 566 can be used to implement arbitration of multi-functional control amounts.

[0202] The actuator 570 can be used to implement the control instructions issued by the control subsystem, and achieve the expected steering, acceleration, and deceleration.

[0203] It should be noted that the above Figure 5 The shown autonomous driving system 500 is only an example rather than a limitation. The autonomous driving system 500 may include more or fewer systems, modules, sub-models, or units, and this application embodiment does not limit this.

[0204] In the prior art, the road structure recognition module 544 in the above Figure 5 can perform lane inference through various solutions, specifically as follows:

[0205] Solution 1:

[0206] Perform lane inference based on the detection of lane markings, that is, detect the lane markings in the image through methods such as machine vision or image detection, and determine the lane to be traveled (or the lane that can be traveled) by the autonomous driving vehicle based on the detected lane markings.

[0207] However, this method overly relies on the actual lane markings and real-time environmental conditions.

[0208] For example, under the current technical conditions, lane marking detection is prone to failure in the following several situations:

[0209] 1. Lane lines are worn or missing;

[0210] 2. Poor lighting conditions, such as backlighting or light and shadow interference like railing shadows;

[0211] 3. Complex road markings, such as having text or guiding marking lines at ramps.

[0212] Solution 2:

[0213] Perform lane inference based on a high-precision map, that is, query the high-precision map to obtain the lane marking line information pre-stored in the high-precision map, and determine the lane to be traveled (or the lane that can be traveled) by the autonomous driving vehicle based on this lane marking line information.

[0214] However, this method overly relies on the production, update, and maintenance of the high-precision map, and also requires relatively high-precision positioning to obtain accurate and reliable lane information.

[0215] Solution 3:

[0216] To make up for the deficiencies in Solution 1 and Solution 2 above, a method of lane inference based on environmental vehicles is proposed in Solution 3, that is, screen for interested vehicles according to preset conditions. After screening out the interested vehicles, determine the driving trajectory of the interested vehicle, and determine the lane to be traveled (or the lane that can be traveled) by the autonomous driving vehicle based on the driving trajectory of the interested vehicle. Among them, the preset conditions may include the distance range of the vehicle, the curvature of the driving trajectory of the vehicle, the observation period of the vehicle, etc.

[0217] However, this method is too dependent on environmental vehicles (i.e., interested vehicles). For example, when there is no vehicle in front, this solution cannot be implemented; when the driving behavior of environmental vehicles does not comply with traffic rules (for example, environmental vehicles do not drive according to the instructions of lane marking lines), the lane to be traveled determined based on the driving trajectory of this environmental vehicle may conflict with the driving trajectories of other vehicles, and thus may bring driving risks.

[0218] Solution 4:

[0219] A solution for end-to-end autonomous driving by combining real-time perception and driving intention, that is, first, in an offline environment, train a neural network model based on the collected training data to obtain a neural network model that can be used for end-to-end autonomous driving; next, in the online session, use local perception (i.e., local environment perception) and driving intention as inputs, and adopt the neural network model trained in the offline environment to calculate the control amount for controlling the vehicle (i.e., controlling the autonomous driving vehicle).

[0220] However, through research and analysis, it is found in this application that the local perception in this method (Solution 4) only includes the image information collected by the camera. The generalization ability for light conditions, road surface types, etc., as well as road surface cognition, the matching of the road surface and driving intention, and the processing of dynamic and static obstacles all need to be processed by the same neural network model, which will lead to the neural network model (for end-to-end autonomous driving) being too complex.

[0221] At the same time, the input of this method is the image information (and driving intention) collected by the camera, and the output is the vehicle control result (for example, the vehicle control result can be the control quantity of the autonomous vehicle). The trained neural network model has a strong dependence on the camera for collecting images and the size of the vehicle to be controlled (i.e., the autonomous vehicle). For example, if the installation position of the camera or the vehicle is changed, it is necessary to re-collect the training data and train the neural network model based on the re-collected training data. Moreover, since the output of this method is the vehicle control result and does not output the lane to be traveled (or the lane that can be traveled) by the autonomous vehicle, the decision-making control logic during autonomous driving using this method cannot be traced.

[0222] Therefore, this application proposes a method for inferring lanes. By obtaining traffic element data detected by a variety of different types of sensors located in the same coordinate system and performing lane inference based on this traffic element data, the lane of the vehicle can be obtained, which helps to reduce the complexity of the neural network model for lane inference.

[0223] The following combines Figures 6 to 16 to elaborate in detail on the method for inferring lanes in the embodiments of this application.

[0224] Figure 6 is a schematic structural diagram of a road structure cognition module 600 provided in an embodiment of this application.

[0225] Figure 6 The road structure cognition module 600 in Figure 5 can be the road structure cognition module 544 in the above

[0226] such as Figure 6 shown, this road structure cognition module 600 can include a road structure modeling sub-module 610 for sensing, a road structure modeling sub-module 620 for map roads, a road structure modeling sub-module 630 for combining environment and intention, and a road model reliability analysis sub-module 640.

[0227] The road structure modeling sub-module 610 for perception can be used to implement Solution 1 in the aforementioned prior art, that is, lane inference based on lane marking lines. The main application scenarios of this module 610 can be where the lane marking lines are clearly visible and the line type, curvature, etc. of the lane marking lines are within the perception range capabilities of (a machine vision model or an image detection model).

[0228] The road structure modeling sub-module 620 for map can be used to implement Solution 2 in the aforementioned prior art, that is, lane inference based on a high-precision map. The main application scenarios of this module 620 can be where high-precision positioning and a high-precision map are available.

[0229] The road structure modeling sub-module 630 for combining environment and intention can be used to implement the method for inferring lanes in the embodiments of the present application. This method does not absolutely rely on lane marking lines, high-precision positioning, or a high-precision map. Therefore, the application of this module 630 can be for all scenarios.

[0230] The road model reliability analysis sub-module 640 can be used to analyze the road models output by the above-mentioned module 610, module 620, and module 630 (for example, the lanes to be traveled or the lanes that can be traveled by an autonomous vehicle), evaluate the road models output by these several modules, determine the reliability levels of each road model, and output the road model with the highest reliability level and the reliability level of this road model.

[0231] It should be noted that the lanes to be traveled (or the lanes that can be traveled) by the autonomous vehicle in the embodiments of the present application can be the shape information of the lanes, for example, the geometric shape of the lanes, or can also be the cost information of the lanes, for example, the cost map (or cost grid map) of the lanes, or can also be other information that can describe the lanes. The embodiments of the present application do not limit this.

[0232] Figure 7 It is a schematic flowchart of the method 700 for inferring lanes provided by the embodiments of the present application. Figure 7 The method 700 shown can be executed by Figure 1 the processor 113 in the vehicle 100, or can also be executed by Figure 2 the processor 103 in the autonomous driving system, or can also be executed by Figure 4 the processor 320 in the server 320.

[0233] Figure 7 The method 700 shown can include step 710 and step 720. It should be understood that Figure 7 the method 700 shown is only an example and not a limitation. The method 700 can include more or fewer steps. The embodiments of the present application do not limit this. The following will introduce these several steps in detail.

[0234] S710. Obtain the detection results of multiple different types of sensors of the vehicle.

[0235] Among them, the multiple different types of sensors may include cameras, lidars or millimeter-wave radars. Alternatively, the multiple different types of sensors may also include other sensors, which are not limited in the embodiments of the present application.

[0236] S720. Process the detection results to obtain traffic element data in the same coordinate system.

[0237] Among them, the traffic element data may be data in the same coordinate system.

[0238] Optionally, the traffic element data may be the type, contour, position and / or motion trajectory of a traffic element. The traffic element may be a stationary object, or the traffic element may also be a moving object.

[0239] For example, the traffic element may include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads (such as roads or curbs around the vehicle), or traffic lights.

[0240] It should be understood that obtaining the traffic element data in the same coordinate system may include: directly obtaining the original data detected by the sensor. In this case, the detected original data is the traffic element data.

[0241] Or, obtaining the traffic element data in the same coordinate system may also include: after obtaining the detection results detected by the sensor (such as the original data or original image obtained by the sensor), processing the detected detection results to obtain a processing result. In this case, the processing result obtained after processing is the traffic element data.

[0242] It should be noted that the processing of the detected detection results (such as the detection results of the sensor) may refer to S810 in method 800 below. Figure 8 in

[0243] Specifically, it may include at least one of the following multiple processing methods for the detection results:

[0244] Processing method 1:

[0245] Perform coordinate system conversion on the detection results (such as the original data or original image obtained by the sensor).

[0246] For example, the ranging data detected by lidar can be converted to a specified coordinate system, or the ranging data can be converted between multiple coordinate systems. The coordinate systems mentioned here may include a vehicle body coordinate system, a relative coordinate system, an image coordinate system, etc., or may include other common coordinate systems, which are not limited herein.

[0247] Processing method 2:

[0248] Determine (or identify) the type of traffic elements included in the detection result (for example, lane marking lines, other vehicles around the vehicle, pedestrians, roads, traffic lights, etc.). That is to say, it can be considered that the traffic element data includes the type of traffic elements.

[0249] For example, the traffic elements included in the image captured by the camera can be identified.

[0250] Processing method 3:

[0251] Based on the detection result, determine the outline, position, and / or movement trajectory of the traffic element (corresponding to the detection result).

[0252] For example, existing algorithms can be used to process the detection result (or the traffic elements included in the detection result of the sensor) to determine the outline, position, and / or movement trajectory of the traffic element.

[0253] Among them, the outline of the traffic element may include geometric features of the traffic element or the center line of the traffic element, etc., which can describe the outline features of the traffic element. This is not limited in the embodiments of the present application.

[0254] The specific implementation methods of the above multiple processing methods can refer to the prior art and will not be elaborated in the embodiments of the present application.

[0255] It should be understood that the above method for processing the detected detection result (for example, the detection result of the sensor) is only an example and not a limitation. Other processing can also be performed on the detection result (for example, filtering the movement trajectory of the traffic element), which is not limited in the embodiments of the present application.

[0256] In the embodiments of the present application, the method 700 may further include S712.

[0257] S712. Determine an environmental perception image according to the traffic element data.

[0258] Among them, the environmental perception image can be used to indicate the environment around the vehicle, and the environmental perception image may include at least one of a surround view image or a front view image.

[0259] For example, the environment perception image may be a world model. In the field of autonomous driving, the world model may refer to a summary of the environment perception information of the vehicle.

[0260] Optionally, based on the traffic element data, the traffic elements may be superimposed on the environment perception image.

[0261] For example, the traffic element data may be converted to the same coordinate system, and then according to the position and perspective of each traffic element relative to the vehicle, the traffic element data may be superimposed on the environment perception image, that is, the panoramic view image or the front view image of the vehicle is obtained.

[0262] The specific stitching algorithm may refer to the prior art and will not be elaborated in the embodiments of the present application.

[0263] S730, using a preset neural network model, perform lane inference based on the traffic element data to obtain the lane of the vehicle.

[0264] Optionally, when determining the environment perception image according to the traffic element data in S720 above, a preset neural network model may also be used to perform lane inference based on the environment perception image to obtain the lane of the vehicle.

[0265] Among them, the neural network model may be obtained after being trained based on the (offline collected) traffic element data and / or driving intention. The specific training method may refer to the prior art and will not be elaborated here.

[0266] Optionally, the driving intention of the vehicle may be a route schematic diagram of the lane to be traveled by the vehicle or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane or driving intersection of the vehicle.

[0267] For example, the semantic indication may include "turn left", "turn right", "go straight" and "stop"; or, the semantic indication may also include "take the two lanes on the right", "take the leftmost lane" and "exit from the second exit", etc.

[0268] It should be understood that the above examples are only illustrative and not restrictive, and the embodiments of the present application do not limit the manner (or form) of the semantic indication.

[0269] At the same time, the embodiments of the present application do not limit the method for determining the driving intention, and the specific method may refer to the prior art and will not be elaborated here.

[0270] In an embodiment of the present application, a preset neural network model may also be used to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel on.

[0271] Optionally, the lane of the vehicle may be represented by the center line of the lane and the width of the lane; or, the lane of the vehicle may also be represented by a road surface cost map of the lane of the vehicle; or, the lane of the vehicle may also be represented in other ways in the prior art, and the embodiments of the present application do not limit this.

[0272] Figure 8 It is a schematic flowchart of a method 800 for inferring a lane provided by an embodiment of the present application. Figure 8 The method 700 shown may be executed by the processor 113 in the vehicle 100 in Figure 1 ; or, it may also be executed by the processor 103 in the autonomous driving system in Figure 2 ; or, it may also be executed by the processor 320 in the server 320 in Figure 4 .

[0273] Figure 8 The method 800 shown may include step 810, step 820, step 830, and step 840. It should be understood that Figure 8 the method 800 shown is only an example and not a limitation. The method 800 may include more or fewer steps, and the embodiments of the present application do not limit this. The following will introduce these steps in detail respectively.

[0274] S810, process the detection result of the sensor to obtain traffic element data.

[0275] The processing of the detection result of the sensor mentioned here includes: performing coordinate system conversion on the detection result of the sensor (for example, the raw data or raw image obtained by the sensor, etc.), determining (or identifying) the traffic elements included in the detection result of the sensor, and determining at least one of the contour, position, and / or movement trajectory of the traffic element (corresponding to the detection result) based on the detection result of the sensor.

[0276] Specifically, reference may be made to Figure 7 the description of various processing methods (performed on the detection result of the sensor) in

[0277] Optionally, S810 above may be as shown in the method 900 in Figure 9 , and this method 900 may include step 910, step 920, and step 930.

[0278] S910, match and splice the road in the detection result with the road at the previous moment (or previous frame).

[0279] Optionally, the detection result can be regarded as the detection result of the current frame. Through S910, the matching and splicing of the road in the current frame (in the detection result) and the road in the previous frame (in the detection result) can be achieved.

[0280] Optionally, through S910, the matching and splicing of the road in the current frame (in the detection result) and the roads in other frames before the current frame (in the detection result) can also be achieved.

[0281] S920, match and splice the lane marking lines in the detection result with the lane marking lines at the previous moment.

[0282] S930, update the trajectory of the moving object in the detection result and perform filtering processing on the trajectory.

[0283] It should be noted that each step in the above method 900 is not a step that must be executed. Only when the corresponding traffic elements are included in the detection result will the corresponding steps be executed.

[0284] For example, when the road is included in the detection result, S910 is executed; otherwise, S910 is not executed.

[0285] S820, determine the environmental perception image according to the traffic element data.

[0286] Optionally, the above S820 can be implemented through any of the following solutions.

[0287] Solution 1:

[0288] As Figure 10 shown, Solution 1 may include step 1010 and step 1020, which are as follows.

[0289] S1010, splice the surround-view images.

[0290] According to the positions and perspectives of the images collected by the sensors of multiple perspectives relative to the vehicle, splice the images collected by the sensors of multiple perspectives to form a complete surround-view image.

[0291] S1020, superimpose the traffic elements onto the surround-view image to obtain the environmental perception image.

[0292] It should be noted that in the above S1020, only when a certain type of traffic element is included will the step corresponding to the traffic element be executed.

[0293] For example, when a road is included in the detection result, the road is superimposed on the surround view image.

[0294] Solution 2:

[0295] As Figure 11 shown, Solution 2 may include Step 1110 and Step 1120, as follows.

[0296] S1110, generate an initial environment image.

[0297] For example, an initial image centered on the vehicle with a blank environment can be generated, and this image can be referred to as the initial environment image.

[0298] S1120, superimpose the traffic elements on the initial environment image to obtain an environment perception image.

[0299] Similar to S1020 above, in S1120, only when a certain type of traffic element is included, the steps corresponding to that traffic element will be executed.

[0300] The environment perception images obtained by the above two solutions may both include a front view image and / or a surround view image. At the same time, the stitching method and the superimposing method in the above two solutions can specifically refer to the prior art and will not be elaborated here.

[0301] S830, determine the driving intention of the vehicle.

[0302] Optionally, S830 can be implemented by any of the following methods.

[0303] Method 1:

[0304] As Figure 12 shown in Method 1200 of

[0305] S1210, determine whether to re-plan the route.

[0306] Optionally, it can be determined whether the vehicle deviates from the planned route according to the current position of the vehicle.

[0307] S1220, obtain the sequence of road points of the road to be traveled.

[0308] If the vehicle has deviated from the planned route, the route can be re-planned to obtain the sequence of road points of the road to be traveled; otherwise, the original sequence of road points of the road to be traveled can be continued to be used, that is, the original sequence of road points of the road to be traveled is used as the sequence of road points of the road to be traveled.

[0309] The sequence of waypoints of the to-be-traveled road can be represented by {P(1), P(2), P(3), …, P(N)}, where P(k) represents the position information of the k-th waypoint, and both k and N are positive integers.

[0310] S1230, to obtain the waypoint P(i) corresponding to the vehicle currently.

[0311] Optionally, the current position of the vehicle can be projected onto the sequence of waypoints of the to-be-traveled road determined in S1220, and the point in the sequence of waypoints of the to-be-traveled road that is closest to the current position of the vehicle is used as the waypoint P(i) corresponding currently, where i is a positive integer.

[0312] S1240, to estimate the waypoint P(j) corresponding to the vehicle after time t.

[0313] Optionally, the waypoint P(j) corresponding to the vehicle after time t can be estimated according to the current speed of the vehicle, where t and j can be positive integers.

[0314] Among them, the value of t can be preset. For example, t can be set to 10 seconds; or, the value of t can be determined according to the position of the vehicle and the road it travels on, etc. For example, whether the vehicle is located in a crowded section of the city, or whether the vehicle is traveling on a highway.

[0315] S1250, to calculate the change in the orientation angle of the vehicle according to P(i) and P(j).

[0316] S1260, to determine the driving intention according to the change in the orientation angle of the vehicle.

[0317] Optionally, the driving intention determined by Method 1 can be a semantic indication (for example, a discrete driving intention), and can include two types:

[0318] (1) Road-level driving intention

[0319] The road-level driving intention can correspond to road-level navigation and can include "turn left", "turn right", "go straight", "stop", etc.;

[0320] (2) Lane-level driving intention

[0321] The lane-level driving intention can correspond to lane-level navigation and can include the to-be-traveled lane number, such as "the first lane on the left", "two lanes on the right".

[0322] Method 2:

[0323] S1310, whether to re-plan the route.

[0324] Optionally, it can be determined whether the vehicle deviates from the planned route according to the current position of the vehicle.

[0325] S1320. Obtain the sequence of waypoints of the road to be traveled.

[0326] If the vehicle has deviated from the planned route, the route can be replanned to obtain the sequence of waypoints of the road to be traveled; otherwise, the original sequence of waypoints of the road to be traveled can be continued to be used, that is, the original sequence of waypoints of the road to be traveled is used as the sequence of waypoints of the road to be traveled.

[0327] The sequence of waypoints of the road to be traveled can be represented by {P(1), P(2), P(3), …, P(N)}, where P(k) represents the position information of the k-th waypoint, and both k and N are positive integers.

[0328] S1330. Obtain the waypoint P(i) corresponding to the vehicle currently.

[0329] Optionally, the current position of the vehicle can be projected into the sequence of waypoints of the road to be traveled determined in S1220, and the point closest to the current position of the vehicle in the sequence of waypoints of the road to be traveled is used as the current corresponding waypoint P(i), where i is a positive integer.

[0330] S1340. Determine the local map around the waypoint P(i).

[0331] Optionally, the road network map within a radius D around the waypoint P(i) currently corresponding to the vehicle can be used as the local map.

[0332] Among them, D is a positive integer, and the value of D can be determined according to the current speed of the vehicle. For example, when the vehicle speed is low, the value of D can be 20 meters.

[0333] S1350. Estimate the waypoint P(i+m) corresponding to the vehicle after time t.

[0334] Optionally, the waypoint P(i+m) corresponding to the vehicle after time t can be estimated according to the current speed of the vehicle, where t, i, and m are all positive integers.

[0335] Among them, the value of t can be preset. For example, t can be set to 10 seconds; or, the value of t can be determined according to the position of the vehicle and the road it travels on, etc. For example, whether the vehicle is in a crowded section of the city, or whether the vehicle is traveling on a highway.

[0336] S1360. Estimate the waypoint P(i-n) corresponding to the vehicle before time t.

[0337] Optionally, the waypoint P(i-n) corresponding to the vehicle before time t can be estimated according to the current speed of the vehicle, where n is a positive integer.

[0338] S1370. Draw a local map based on waypoints P(i - n), P(i), and P(i + m).

[0339] Optionally, {P(i), …, P(i + m)} can be drawn as waypoints to be traveled on the local map, and {P(i - n), …, P(i)} can be drawn as historical waypoints (i.e., waypoints already traveled) on the local map. Among them, the markings of the waypoints to be traveled on the local map are different from the markings of the historical waypoints on the local map. For example, the colors or types of the markings are different.

[0340] As Figure 14 shown, the thin solid line represents the waypoints to be traveled, the dashed line represents the historical waypoints, and the thick solid line represents the road.

[0341] S1380. Determine the driving intention of the vehicle based on the local map.

[0342] Optionally, the driving intention determined by Method 2 can be a route schematic diagram (e.g., a local route schematic diagram). The route schematic diagram can include the historical path and the path to be traveled of the vehicle. For example, Figure 14 the three route schematic diagrams A, B, and C shown in

[0343] In Figure 14 , part A represents the straight - ahead schematic diagram on a straight road, and the path to be traveled is a straight - ahead route; part B represents that the vehicle has traveled a right - turn route, and the path to be traveled is a straight - ahead route; part C represents that the vehicle has traveled a straight - ahead route, and the route to be traveled is a right - turn route.

[0344] S840. Use a preset neural network model to perform lane inference based on the environmental perception image and the driving intention to obtain the lane of the vehicle.

[0345] Among them, the lane of the vehicle can be expressed in any of the following ways:

[0346] Method 1:

[0347] Lane centerline and width: Set the length requirement of the lane according to the environment where the vehicle is located (i.e., the field of view of the vehicle and system requirements).

[0348] For example, when the vehicle is in an environment with a slow speed, the length requirement of the lane can be 20 m. If sampling is carried out at a distance of 1 m per point, 20 position points are needed to represent the lane. Among them, each position point includes three values, corresponding to the abscissa, ordinate, and width of the position point respectively.

[0349] Method 2:

[0350] Cost grid map: The area around the vehicle can be divided into P*Q grids, and the lane of the vehicle can also be represented by multiple grids among the P*Q grids, where both P and Q are positive integers.

[0351] According to the cognitive ability of in-vehicle sensors, the resolution of the cost grid map, that is, the size of each grid, can be designed as 20cm*20cm or 10cm*10cm. Each grid has a value in the range of [0,1] to represent its cost value.

[0352] Optionally, the neural network model can be obtained after being trained based on environmentally perceived images and driving intentions collected offline.

[0353] Optionally, in S840, the environmentally perceived image determined in S820 and the driving intention determined in S830 can be used as inputs, and the pre-trained neural network model can be used for lane inference to obtain the lane of the vehicle.

[0354] For example, as Figure 15 shown, the environmentally perceived image determined by Solution 1 in S820 and the driving intention determined by Method 1 in S830 can be used as inputs, and the output is the center line of the lane and the width of the lane. The structure of the neural network model can be as Figure 15 shown.

[0355] As Figure 15 shown, the neural network model can use the ResNet50 network to extract features from the environmentally perceived image to obtain environmentally perceived features, and can use a decoder to decode the driving intention into driving intention features (i.e., discrete driving intentions). The driving intention features can use four-dimensional vectors to respectively correspond to the four discrete driving intentions of "turn left", "turn right", "go straight", and "stop"; the neural network model can use a concatenate layer to connect the environmentally perceived features and the driving intention features to obtain the connected features, and input the connected features into a fully connected layer; the fully connected layer can generate 4 parallel fully connected models (the "4" here is the number of discrete driving intentions). Among them, each fully connected model has two compression layers to highly compress the features. Finally, the depth of the features output by the fully connected layer (i.e., 4 parallel fully connected models) drops to 4. At the same time, dropout can be performed once before each compression layer to prevent feature overfitting by masking some connection elements; finally, the neural network model selects the predicted output corresponding to the input intention from the features output by the fully connected layer (i.e., 4 parallel fully connected models), that is, one of the fully connected models, and this fully connected model is the inferred lane.

[0356] For another example, as Figure 16As shown in the figure, the environmental perception image determined in the second solution of S820 and the driving intention determined in the second method of S830 can be used as inputs, and the output is the center line of the lane and the width of the lane. The structure of the neural network model can be as Figure 16 shown.

[0357] As Figure 16 shown, the neural network model can use the ResNet50 network to extract features from the environmental perception image to obtain environmental perception features. Similarly, the ResNet50 network can also be used to extract features from the driving intention to obtain driving intention features;

[0358] The neural network model uses a concatenate layer to connect the environmental perception features and the driving intention features to obtain the concatenated features, and inputs the concatenated features into the fully connected layer;

[0359] The fully connected layer has two compression layers, which can highly compress the features. Finally, the depth of the features output by the fully connected layer is reduced to 1. At the same time, dropout can be performed once before each compression layer to prevent feature overfitting by masking some connection elements; finally, the fully connected layer outputs the inferred lane.

[0360] It should be noted that the processing process of the neural network model in the above Figure 15 and Figure 16 can specifically refer to the prior art and will not be elaborated in the embodiments of the present application.

[0361] Figure 17 is a schematic flowchart of method 1700 for training a lane inference model provided by an embodiment of the present application. Figure 17 The method 1700 shown in the figure can be executed by the processor 113 in the vehicle 100 in Figure 1 , or can also be executed by the processor 103 in the autonomous driving system in Figure 2 , or can also be executed by the processor 320 in the server 320 in Figure 4 .

[0362] Figure 17 The method 1700 shown in the figure may include step 1710, step 1720 and step 1730. It should be understood that Figure 17 the method 1700 shown in the figure is only an example and not a limitation. The method 1700 may include more or fewer steps, which are not limited in the embodiments of the present application. The following will introduce these steps in detail respectively.

[0363] S1710, obtaining the detection results of various different types of sensors of the vehicle.

[0364] S1720. Perform coordinate system conversion on the detection results to obtain traffic element data in the same coordinate system.

[0365] Among them, the traffic element data can be data in the same coordinate system.

[0366] S1730. Use a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.

[0367] S1740. Adjust the parameters of the lane inference model according to the lane of the vehicle.

[0368] Optionally, obtaining traffic element data detected by a variety of different types of sensors of the vehicle may include: obtaining the detection results of the variety of different types of sensors; processing the detection results to obtain the traffic element data.

[0369] Optionally, processing the detection results to obtain the traffic element data may include: performing coordinate system conversion on the detection results to obtain the traffic element data.

[0370] Optionally, the variety of different types of sensors may include a camera, a lidar, or a millimeter wave radar.

[0371] Optionally, the traffic element data may be the contour, position, and / or movement trajectory of the traffic element, and the traffic element may include one or more of lane marking lines, other vehicles, pedestrians, roads, or traffic lights around the vehicle.

[0372] Optionally, the method may further include: determining an environment perception image according to the traffic element data, the environment perception image being used to indicate the environment around the vehicle and the traffic element, and the environment perception image including at least one of a panoramic image or a front view image; among them, using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle may include: using the lane inference model to perform lane inference based on the environment perception image to obtain the lane of the vehicle.

[0373] Optionally, using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle may include: using the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle, the driving intention of the vehicle being used to indicate the route to be traveled by the vehicle.

[0374] Optionally, the driving intention of the vehicle may be a schematic diagram of the route of the lane to be traveled by the vehicle or a vehicle semantic indication, and the vehicle semantic indication may be used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0375] Optionally, the lane of the vehicle may be represented by the center line of the lane and the width of the lane, or the lane of the vehicle may be represented by a road surface cost map.

[0376] Figure 18 It is a schematic block diagram of a device 1800 for inferring a lane provided by an embodiment of the present application. It should be understood that Figure 18 The shown device 1800 for inferring a lane is only an example, and the device 1800 of the embodiment of the present application may further include other modules or units. It should be understood that the behavior planning device 1800 can execute Figure 7 or Figure 8 each step in the method of, and for the sake of avoiding repetition, details are not described herein again.

[0377] An acquisition unit 1810 is configured to acquire detection results of a plurality of different types of sensors of the vehicle.

[0378] A conversion unit 1820 is configured to perform coordinate system conversion on the detection results to obtain traffic element data in the same coordinate system.

[0379] A lane inference unit 1830 is configured to use a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.

[0380] Optionally, the acquisition unit 1810 is specifically configured to: acquire the detection results of the plurality of different types of sensors; process the detection results to obtain the traffic element data.

[0381] Optionally, the acquisition unit 1810 is specifically configured to: perform coordinate system conversion on the detection results to obtain the traffic element data.

[0382] Optionally, the plurality of different types of sensors include a camera, a lidar, or a millimeter wave radar.

[0383] Optionally, the traffic element data is the contour, position, and / or movement trajectory of a traffic element, and the traffic element includes one or more of lane marking lines, other vehicles, pedestrians, roads, or traffic lights around the vehicle.

[0384] Optionally, the device 1800 further includes a superimposing unit 1840 configured to: determine an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a surround view image or a front view image; wherein, the lane inference unit 1830 is specifically configured to: use a preset neural network model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

[0385] Optionally, the lane inference unit 1830 is specifically configured to: use a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

[0386] Optionally, the driving intention of the vehicle is a schematic diagram of the route of the lane that the vehicle is to travel or a vehicle semantic indication, where the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

[0387] Optionally, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0388] It should be understood that the device 1800 for inferring lanes is embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto. For example, a "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0389] As an example, the device 1800 for inferring lanes provided in the embodiments of the present application can be an in-vehicle device of an autonomous vehicle, or can be a chip configured in an in-vehicle device to be used for executing the method described in the embodiments of the present application.

[0390] Figure 19 It is a schematic block diagram of a device 1900 for training a lane inference model provided in an embodiment of the present application. It should be understood that Figure 19 The device 1900 for training a lane inference model shown is only an example, and the device 1900 in the embodiments of the present application may further include other modules or units. It should be understood that the behavior planning device 1900 can execute Figure 17For the steps in the method, to avoid repetition, they will not be elaborated here.

[0391] An acquisition unit 1910, configured to acquire detection results of multiple different types of sensors of a vehicle.

[0392] A conversion unit 1920, configured to perform coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system.

[0393] A lane inference unit 1930, configured to use a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle;

[0394] An adjustment unit 1940, configured to adjust parameters of the lane inference model according to the lane of the vehicle.

[0395] Optionally, the acquisition unit 1910 is specifically configured to: acquire detection results of the multiple different types of sensors; process the detection results to obtain the traffic element data.

[0396] Optionally, the acquisition unit 1910 is specifically configured to: perform coordinate system conversion on the detection results to obtain the traffic element data.

[0397] Optionally, the multiple different types of sensors include a camera, a lidar, or a millimeter-wave radar.

[0398] Optionally, the traffic element data is the contour, position, and / or motion trajectory of a traffic element, and the traffic element includes one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads, or traffic lights.

[0399] Optionally, the device 1900 further includes a superimposing unit 1950, configured to: determine an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic element, and the environmental perception image includes at least one of a panoramic image or a front view image; wherein, the lane inference unit 1930 is specifically configured to: use the lane inference model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

[0400] Optionally, the lane inference unit 1930 is specifically configured to: use the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

[0401] Optionally, the driving intention of the vehicle is a route schematic diagram of the lane to be traveled by the vehicle or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane or driving intersection of the vehicle.

[0402] Optionally, the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

[0403] It should be understood that the device 1900 for training the lane inference model is embodied in the form of a functional module. The term "module" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto. For example, the "module" can be a software program, a hardware circuit or a combination of the two for implementing the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor or a group of processors, etc.) for executing one or more software or firmware programs, a memory, a merged logic circuit and / or other suitable components for supporting the described functions.

[0404] As an example, the device 1900 for training the lane inference model provided in the embodiments of the present application can be an in-vehicle device of an autonomous vehicle, or can be a chip configured in the in-vehicle device for executing the method described in the embodiments of the present application.

[0405] Figure 20 It is a schematic block diagram of the device 800 according to an embodiment of the present application. Figure 20 The illustrated device 800 includes a memory 801, a processor 802, a communication interface 803 and a bus 804. Among them, the memory 801, the processor 802 and the communication interface 803 are communicatively connected to each other through the bus 804.

[0406] The memory 801 can be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 801 can store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 is used to execute the respective steps of the method for planning and inferring lanes or the method for training a lane inference model in the embodiments of the present application. For example, it can execute Figure 7 , Figure 8 or Figure 17 the respective steps of the illustrated embodiments.

[0407] The processor 802 may be a general - purpose central processing unit (CPU), a microprocessor, an application - specific integrated circuit (ASIC), or one or more integrated circuits, which are used to execute relevant programs to implement the method of the inference lane in the method embodiments of the present application or the method of training the lane inference model.

[0408] The processor 802 may also be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the method for planning an autonomous driving vehicle in the embodiments of the present application can be completed by the integrated logic circuit in the hardware of the processor 802 or the instructions in the form of software.

[0409] The above - mentioned processor 802 may also be a general - purpose processor, a digital signal processor (DSP), an application - specific integrated circuit (ASIC), a field - programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general - purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0410] The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as random access memory, flash memory, read - only memory, programmable read - only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory 801. The processor 802 reads the information in the memory 801 and combines its hardware to complete the functions required to be executed by the units included in the behavior planning device of the autonomous driving vehicle in the embodiments of the present application, or execute the method for planning an autonomous driving vehicle in the method embodiments of the present application. For example, it can execute Figure 7 、 Figure 8 or Figure 17 each step / function of the embodiments shown.

[0411] The communication interface 803 may use, but is not limited to, a transceiver - type transceiver device to implement the communication between the device 800 and other devices or communication networks.

[0412] The bus 804 may include a path for transmitting information between various components of the device 800 (for example, the memory 801, the processor 802, the communication interface 803).

[0413] It should be understood that the device 800 shown in the embodiments of the present application may be an in-vehicle device in an autonomous vehicle, or may also be a chip configured in the in-vehicle device.

[0414] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0415] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0416] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that includes one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0417] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. Additionally, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship. The specific meaning can be understood by referring to the context.

[0418] In this application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0419] It should be understood that in various embodiments of the present application, the magnitudes of the sequence numbers of the above processes do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0420] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled professionals may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0421] Those skilled in the art can clearly understand that for the sake of convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0422] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0423] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0424] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0425] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0426] As described above, the above are only specific implementation manners of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A method for inferring a lane, characterized in that, Including: Obtaining the detection results of various different types of sensors of the vehicle; Performing coordinate system conversion on the detection results to obtain traffic element data in the same coordinate system, where the traffic element data is the contour, position, and / or movement trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads, or traffic lights; Determining an environmental perception image based on the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a panoramic view image or a forward view image; Using a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle, including: Using the preset neural network model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

2. The method according to claim 1, wherein The using the preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: Using the preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

3. The method according to claim 2, wherein The driving intention of the vehicle is a schematic diagram of the route of the lane that the vehicle is to travel or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane, or driving intersection of the vehicle.

4. The method according to any one of claims 1 to 3, characterized in that The lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

5. A method for training a lane inference model, characterized in that, Including: Obtaining the detection results of various different types of sensors of the vehicle; Performing coordinate system conversion on the detection results to obtain traffic element data in the same coordinate system, where the traffic element data is the contour, position, and / or movement trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads, or traffic lights; Determining an environmental perception image based on the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a panoramic view image or a forward view image; Using a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle, including: using the lane inference model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle; Adjusting the parameters of the lane inference model according to the lane of the vehicle.

6. The method according to claim 5, characterized in that, The using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: Using the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, where the driving intention of the vehicle is used to indicate the route that the vehicle is to travel.

7. The method according to claim 6, wherein The driving intention of the vehicle is a route schematic diagram of the lane on which the vehicle is to travel or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane or driving intersection of the vehicle.

8. The method according to any one of claims 5 to 7, characterized in that, The lane of the vehicle is represented by the center line and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

9. An apparatus for inferring lanes, characterized in that, Comprising: An acquisition unit for acquiring the detection results of various different types of sensors of the vehicle; A conversion unit for performing coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system, where the traffic element data is the contour, position, and / or motion trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads or traffic lights; An overlay unit for determining an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a panoramic image or a front view image; A lane inference unit for using a preset neural network model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle.

10. The device according to claim 9, wherein Specifically, the lane inference unit is configured to: Use a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route on which the vehicle is to travel.

11. The device according to claim 10, characterized in that, The driving intention of the vehicle is a route schematic diagram of the lane on which the vehicle is to travel or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane or driving intersection of the vehicle.

12. The device according to any one of claims 9 to 11, characterized in that The lane of the vehicle is represented by the center line and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

13. An apparatus for training a lane inference model, characterized in that, Comprising: An acquisition unit for acquiring the detection results of various different types of sensors of the vehicle; A conversion unit for performing coordinate system conversion on the detection results to obtain traffic element data located in the same coordinate system, where the traffic element data is the contour, position, and / or motion trajectory of traffic elements, and the traffic elements include one or more of lane marking lines, other vehicles around the vehicle, pedestrians, roads or traffic lights; An overlay unit for determining an environmental perception image according to the traffic element data, where the environmental perception image is used to indicate the environment around the vehicle and the traffic elements, and the environmental perception image includes at least one of a panoramic image or a front view image; A lane inference unit for using a lane inference model to perform lane inference based on the environmental perception image to obtain the lane of the vehicle; An adjustment unit for adjusting the parameters of the lane inference model according to the lane of the vehicle.

14. The device according to claim 13, characterized in that, Specifically, the lane inference unit is configured to: Use the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route on which the vehicle is to travel.

15. The device according to claim 14, characterized in that, The driving intention of the vehicle is a schematic diagram of the route of the lane in which the vehicle is to travel or a vehicle semantic indication, and the vehicle semantic indication is used to indicate at least one of the driving direction, driving lane or driving intersection of the vehicle.

16. The device according to any one of claims 13 to 15, characterized in that, The lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road surface cost map.

17. A device for inferring lanes, characterized in that, It includes a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 1 to 4.

18. An apparatus for training a lane inference model, characterized in that, It includes a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 5 to 8.

19. A vehicle, characterized in that, It includes the device according to any one of claims 9 to 17.

20. A computer-readable storage medium, characterized in that, Program instructions are stored in the computer-readable storage medium, and when the program instructions are run by a processor, the method according to any one of claims 1 to 8 is implemented.

21. A chip, characterized in that, The chip includes a processor and a data interface. The processor reads the instructions stored on the memory through the data interface to execute the method according to any one of claims 1 to 8.

Images