Intelligent driving method and apparatus

Abstract

An intelligent driving method and apparatus are disclosed. The method includes the steps of: acquiring data collected by vehicle sensors for a scene, wherein the sensors include at least one of a camera and radar; inputting the acquired data into a perception detection network and outputting perception information, wherein the perception information indicates voxels of obstacles in a first scene; and controlling the driving of the vehicle based on at least the perception information. This helps to enhance the vehicle's ability to perceive its surroundings, improve the accuracy of obstacle detection, and avoid collisions.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and in particular, to an intelligent driving method and apparatus.

Background Art

[0002] The perception ability of the perception system of an automated vehicle with respect to its surroundings is closely related to the safe driving of the vehicle.

[0003] Currently, there is a vehicle perception system that detects surrounding obstacles in a pure vision manner. This method has a high dependence on training materials (such as a trust list), and the perception system can only identify obstacles after training and learning of the obstacles.

[0004] Currently, there is also a vehicle perception system that detects surrounding obstacles by using lidar or millimeter-wave radar. However, such detection is easily affected by the weather. For example, during rainy or snowy days, the detection accuracy of such obstacles is low.

Summary of the Invention

Means for Solving the Problems

[0005] This application discloses an intelligent driving method and apparatus that helps to enhance the vehicle's perception ability with respect to surrounding objects, improve the obstacle detection accuracy, and avoid collisions.

[0006] According to a first aspect, this application provides an intelligent driving method. The method includes the steps of obtaining the collected data of a sensor for a first scene, where the sensor includes at least one of a camera and a radar; inputting the collected data into a perception detection network and outputting perception information, where the perception information indicates the voxels of obstacles in the first scene; and controlling the driving of the vehicle based at least on the perception information.

[0007] Here, an obstacle is an entity that is not expected to collide with a vehicle during the driving process. Entities may be static or dynamic. Static entities may be static objects with volume and mass, such as containers on the road, road construction signs, road boundary rails, soil deposits, tires, overturned vehicles, lying people or animals, roadside buildings, trees, parked vehicles, road signs, utility poles, or roadside curbing belts. Dynamic entities may be moving objects with volume and mass, such as pedestrians (such as walking pedestrians or cyclists), animals, vehicles, or vehicles carrying goods (e.g., containers, tree branches, or other items).

[0008] It will be understood that the forms in which obstacles are presented in the physical world are not limited in this application. For example, the obstacle may be a vehicle, and the form in which the vehicle appears may be in the form of its tires being on the ground while it is moving or stopped, in the form of the vehicle overturning after a collision, in the form of items (e.g., tree branches or containers) being loaded in the rear trunk of the vehicle, or in the form of multiple vehicles linked together. The type of vehicle when the obstacle is a vehicle is not limited here. The type of vehicle may be, for example, a car, truck, bus, trailer, incomplete vehicle, motorcycle, or bicycle.

[0009] Here, the radar includes at least one of lidar and millimeter-wave radar.

[0010] For example, the first scene may be understood as the environmental space that can be detected by sensors on the vehicle during the vehicle's driving process. In the vehicle's driving process, each time point may correspond to one scene, or a scene corresponding to multiple times may contain scenes corresponding to each of those multiple times.

[0011] For example, a perception detection network is trained based on a sensor dataset and label information corresponding to the sensor dataset, generated by 4D reconstruction (i.e., spatiotemporal reconstruction including dynamic and static targets). The label information is used in the training process of the perception detection network to provide actual value information for the perception detection network's prediction results. It will be understood that the 4D reconstruction may describe changes in entity objects in three-dimensional space in the time dimension.

[0012] For example, in addition to controlling the driving of the vehicle based on perceptual information, the driving of the vehicle may be further controlled based on at least one of the following: navigation map information, high-definition map information, roadside devices, and live traffic information broadcast by other surrounding vehicles.

[0013] For example, the method may be applied to a vehicle or a component (e.g., a chip or integrated circuit) used for intelligent driving control within a vehicle. The vehicle is configured with an autonomous driving system, where the autonomous driving system is not limited to a fully autonomous driving system, a highly autonomous driving system, a conditionally autonomous driving system, a partially autonomous driving system, etc. Those skilled in the art will understand that all non-fully manual driving systems for intelligent driving may be included in this concept.

[0014] In the aforementioned method, scene data is collected using a purely visual method or a combination of visual and radar methods, and the scene data is processed by outputting perceptual information indicating obstacle voxels using a perceptual detection network. This enhances the vehicle's ability to perceive surrounding objects, enables the perception of obstacles independent of semantic categories, and improves the generalization ability and accuracy of detecting obstacles in the scene. In addition, controlling the vehicle's operation based on perceptual information can improve the vehicle's driving safety.

[0015] In a possible embodiment of the first aspect, the method includes the steps of displaying an obstacle based on perceptual information, wherein the obstacle is marked by a polygon box, and / or displaying voxels of the obstacle based on perceptual information, wherein the voxels of the obstacle are marked by a polygon box.

[0016] For example, a polygon box may be two-dimensional or three-dimensional.

[0017] For example, when displaying obstacles or obstacle voxels, dynamic and static obstacles at the current time may be distinguished by using different colors, or dynamic and static obstacles may be distinguished by displaying additional arrows on dynamic obstacles, where the arrows on dynamic obstacles indicate the direction of movement of the dynamic obstacle.

[0018] In the aforementioned embodiment, obstacles are marked by polygon boxes mounted closer to the shape of the obstacle. By presenting the obstacle and / or its voxels, the user can clearly and intuitively understand the vehicle's perceived status relative to its surroundings at the current moment.

[0019] In a possible embodiment of the first aspect, the perceptual information includes at least one of the following information: the occupancy status of a voxel in a first scene, velocity information of a voxel in a first scene, visibility status of a voxel in a first scene, and corner point information of a polygon box corresponding to an obstacle, the polygon box corresponding to the obstacle being associated with the voxel of the obstacle.

[0020] Here, the visibility status of a voxel may be classified, for example, into "visible" and "invisible." For example, in a scene in which a vehicle is positioned at the present time, if a particular voxel in that scene is not detected by any observational signals from any sensors on the vehicle (including cameras and radar) at the present time, the visibility status of this voxel is invisible; if this voxel is detected by an observational signal from at least one sensor on the vehicle, the visibility status of the voxel is visible.

[0021] Here, the occupation status of a voxel may be classified as, for example, "occupied" or "empty" (i.e., not occupied). For example, in a scene where a vehicle is positioned at the current time, if a physical entity exists in the spatial location in the physical world where the scene is positioned, corresponding to a particular voxel in the scene, then the occupation status of this voxel is occupied. If no physical entity exists in the spatial location in the physical world where the scene is positioned, corresponding to this voxel, then the occupation status of this voxel is empty. It will be understood that air is not a physical entity.

[0022] For example, associating a polygon box corresponding to an obstacle with the obstacle's voxels may be understood as obtaining the corner point information of the polygon box corresponding to the obstacle based on the index information of the obstacle's voxels. The corner point information of the polygon box corresponding to an obstacle may be obtained computationally, for example, by using a convex hull algorithm based on the index information of the obstacle's voxels.

[0023] In the foregoing embodiments, based on the visibility status of voxels, the blind spots of vehicles in the current scene may be known, or based on the occupancy status of voxels, it may be known that the vehicles in the current scene need to avoid the area where "occupied" voxels are arranged in order to avoid collisions, or obstacles in the scene may be quickly arranged based on the corner point information of the polygon boxes corresponding to the obstacles, or the speed information of the obstacles in the current scene may be determined based on the speed information of the voxels and the corner point information of the polygon boxes corresponding to the obstacles.

[0024] In a possible implementation of the first aspect, the perception information further indicates the voxels of the road surface in the first scene, and the step of controlling the driving of the vehicle based at least on the perception information includes the step of generating the road surface shape information of the first scene based at least on the perception information and the step of adjusting the suspension in the vehicle based on the road surface shape information.

[0025] For example, the road surface shape information indicates the status of the road surface in the first scene (for example, whether the road surface has depressions or bumps).

[0026] In the foregoing embodiments, the vehicle may pre-acquire the status of the road surface in front of the vehicle based on the perception information. When it is detected that the road surface in front is changing, the vehicle has sufficient time to timely adjust the suspension of the vehicle, so the vehicle maintains a horizontal and stable state as much as possible in the driving process, reduces the vibration caused by the change of the road surface, and improves the riding comfort of the vehicle.

[0027] In a possible implementation of the first aspect, the step of controlling the driving of the vehicle based at least on the perception information is the step of adjusting the driving route of the vehicle based at least on the perception information, and the adjusted driving route does not pass through the area where the voxels of the obstacle are arranged.

[0028] In the foregoing embodiments, the vehicle can be prevented from colliding with obstacles during the driving process so as to help improve the driving safety of the vehicle.

[0029] In a possible implementation of the first aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network.

[0030] The image feature extraction network is configured to extract 3D image features of the image data.

[0031] The point cloud feature extraction network is configured to extract point cloud features of voxels corresponding to the point cloud data.

[0032] The feature fusion network is configured to perform fusion based on the 3D image features and the point cloud features of the voxels corresponding to the point cloud data to obtain the fusion features of the voxels of the first scene.

[0033] The output network is configured to process the fusion features of the voxels of the first scene and output perception information.

[0034] Here, different from the voxels corresponding to the point cloud data, in this application, the voxels of the first scene are the voxels obtained by the fusion by the feature fusion network.

[0035] In the foregoing embodiments, the original data of the multimodal sensors on the vehicle is used to perceive surrounding obstacles, and the advantages of different sensors are integrated (for example, the texture semantic information of the image is provided, and the depth information of the point cloud is provided). This helps to enhance the vehicle's perception ability of the surroundings and improve the generalization ability and accuracy of detecting obstacles.

[0036] In a possible embodiment of the first aspect, the method further includes the steps of inputting text query information and fused voxel features of an obstacle into an attribute recognition network and outputting category information of an obstacle, wherein the text query information is used to request a category query; and displaying the category information of an obstacle, wherein the fused voxel features of an obstacle are determined based on corner point information of a polygon box corresponding to the obstacle and the fused voxel features of a first scene, and the polygon box corresponding to the obstacle is associated with the voxel of the obstacle.

[0037] For example, text query information is used to request a query for Q categories. Assume that the number of categories of obstacles in a scene is P, Q and P are positive integers, and Q is greater than P. That is, the number of categories actually supported by the attribute recognition network for recognition is greater than the number of categories of obstacles in any given scene. In this way, it can be guaranteed that omissions can be avoided when the attribute recognition network performs category recognition for obstacles in any given scene.

[0038] For example, determining the voxel fusion features of an obstacle based on the corner point information of the polygon box corresponding to the obstacle and the voxel fusion features of the first scene means that the corner point information of the polygon box corresponding to the obstacle corresponds to the voxel index information of the obstacle, and therefore, the voxel fusion features of the obstacle may be determined from the voxel fusion features of the first scene based on the voxel index information of the obstacle.

[0039] Here, the association of a polygon box corresponding to an obstacle with the obstacle's voxel means that the corner point information of the polygon box corresponding to the obstacle is obtained based on the index information of the obstacle's voxel.

[0040] For example, attribute recognition includes a text coding network and an attribute decoding network. The text coding network is configured to extract word vector features from text query information, and the attribute decoding network is configured to output obstacle category information based on the word vector features and the fused features of the obstacle voxels.

[0041] In the aforementioned embodiment, an attribute recognition network is deployed based on the perception detection network, and as a result, the vehicle can not only detect obstacles in its surroundings but also recognize the category of the obstacles during the driving process, thereby enabling the vehicle to not only see objects but also understand them.

[0042] In a possible embodiment of the first aspect, the method includes the steps of: acquiring a plurality of planned routes for a vehicle; inputting the plurality of planned routes for the vehicle and the fused voxel features of a first scene into a route evaluation network and outputting recommendation coefficients for the plurality of planned routes and a recommended route in the plurality of planned routes, wherein the recommended route is associated with the recommendation coefficients for the plurality of planned routes; and displaying the recommended route.

[0043] For example, a route evaluation network includes a route coding network, a feature interaction network, and an evaluation output network. The route coding network is configured to extract the route features of each of several planned routes. The feature interaction network is configured to obtain the risk features of each planned route based on the route features of each planned route and the fused features of the voxels in the first scene. The evaluation output network is configured to output recommendation coefficients for the multiple planned routes and the recommended route for the multiple planned routes based on the risk features of the multiple planned routes.

[0044] For example, the recommended route is the route that corresponds to the highest recommendation coefficient among multiple planned routes.

[0045] For example, the recommendation factor for a planned route may be obtained based on at least one of the following: the risk factor of the planned route, comfort, and traffic efficiency. The risk factor of the planned route is related to at least one of the following factors: the distance between the planned route and obstacles (including visible obstacles and obstacles currently in blind spots), and whether the planned route collides with the routes of other traffic participants on the road (e.g., whether a collision occurs at the present or future time). The traffic efficiency of the planned route is related to at least one of the following factors: the length of the planned route, the estimated traffic duration corresponding to the planned route, the number of traffic lights on the planned route, and the area of ​​the drivable region on which the planned route is located. The comfort of the planned route is related to at least one of the following factors: the values ​​of the steering acceleration and steering frequency of the planned route, the rate of change of the acceleration of the planned route, the flatness of the road surface of the planned route, the number of traffic lights on the planned route, the type of road on which the planned route is located, and whether the route area of ​​the planned route is cold.

[0046] For example, assuming no other factors change, a lower risk coefficient for a planned route indicates a higher recommendation coefficient for that route. Assuming no other factors change, higher comfort for a planned route indicates a higher recommendation coefficient for that route. Assuming no other factors change, higher traffic efficiency for a planned route indicates a higher recommendation coefficient for that route.

[0047] In the embodiments described above, an attribute recognition network may be deployed based on the perception detection network to perform route recommendation, which helps improve the driving safety and comfort of the vehicle.

[0048] According to a second aspect, the present application provides a system for intelligent driving. The system comprises a perception detection network configured to output perception information based on sensor data collected for a first scene, wherein the perception information indicates voxels of obstacles in the first scene, and the sensor includes at least one of a camera and a radar, and an attribute recognition network configured to output obstacle category information based on text query information and fused features of obstacle voxels, wherein the fused features of obstacle voxels are determined based on corner point information of a polygon box corresponding to the obstacle and fused features of voxels in the first scene, and the fused features of obstacles The ligon box comprises an attribute recognition network, where the fused features of the voxels in a first scene are associated with the voxels of an obstacle, and the fused features of the voxels in a first scene are acquired by a perception detection network by performing temporal and / or spatial fusion based on at least one of the 3D image features and point cloud features of the voxels extracted from the collected data; and a path evaluation network configured to output recommendation coefficients for multiple plan paths and recommended paths in multiple plan paths based on multiple plan paths and the fused features of the voxels in a first scene, where the recommended paths are associated with the recommendation coefficients for multiple plan paths.

[0049] For example, the system may be deployed in a vehicle or in a component used for intelligent driving control within a vehicle, and the component may be, for example, a chip or an integrated circuit. For details of the vehicle, please refer to the description of the vehicle in the first embodiment. Details will not be described again.

[0050] In the aforementioned methods, the perception detection network may be used to enhance the system's perception capabilities for intelligent driving of its surroundings, so that the deployment side of the system can be prevented from colliding with obstacles, thereby improving the safety of the system. The attribute recognition network is used to improve the system's intelligence, so that the system can further recognize obstacle categories based on its perception of obstacles. The path evaluation network can be used to recommend low-risk paths to facilitate intelligent movement.

[0051] For any beneficial effect of any feature in the second embodiment, please refer to the description of the beneficial effect of the corresponding feature in the first embodiment. Further details will not be provided again.

[0052] In a possible embodiment of the second aspect, the perceptual information includes at least one of the following information: the occupancy status of a voxel in a first scene, velocity information of a voxel in a first scene, visibility status of a voxel in a first scene, and corner point information of a polygon box corresponding to an obstacle, the polygon box corresponding to the obstacle being associated with the voxel of the obstacle.

[0053] In a possible embodiment of the second aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network.

[0054] The image feature extraction network is configured to extract 3D image features from image data.

[0055] The point cloud feature extraction network is configured to extract point cloud features from voxels corresponding to the point cloud data.

[0056] The feature fusion network is configured to perform fusion based on 3D image features and point cloud features of voxels corresponding to point cloud data to obtain the fused features of voxels in the first scene.

[0057] The output network is configured to process the fused voxel features of the first scene and output perceptual information.

[0058] In a possible embodiment of the second aspect, the attribute recognition network includes a text coding network and an attribute decoding network, wherein the text coding network is configured to extract word vector features of text query information, and the attribute decoding network is configured to output obstacle category information based on the word vector features and the fused features of the obstacle voxels.

[0059] In a possible embodiment of the second aspect, the route evaluation network includes a route coding network, a feature interaction network, and an evaluation output network.

[0060] The route coding network is configured to extract route features from each of multiple planned routes.

[0061] The feature interaction network is configured to acquire the risk features of each planned path based on the path features of each planned path and the fused voxel features of the first scene.

[0062] The evaluation output network is configured to output recommendation coefficients for multiple planned routes and recommended routes for multiple planned routes, based on the risk characteristics of multiple planned routes.

[0063] According to a third aspect, the present application provides an apparatus for intelligent driving. The apparatus includes a receiving unit configured to acquire sensor data for a first scene, the receiving unit comprising at least one of a camera and a radar, and a processing unit configured to input the acquired data to a perception detection network and output perception information, the perception information indicating voxels of obstacles in the first scene. The processing unit is further configured to control the driving of a vehicle based at least on the perception information.

[0064] In a possible embodiment of the third aspect, the device further includes a display unit, which is configured to display obstacles marked with polygon boxes and / or display voxels of obstacles based on perceptual information.

[0065] In a possible embodiment of the third aspect, the perceptual information includes at least one of the following information: the occupancy status of a voxel in a first scene, velocity information of a voxel in a first scene, visibility status of a voxel in a first scene, and corner point information of a polygon box corresponding to an obstacle, the polygon box corresponding to the obstacle being associated with the voxel of the obstacle.

[0066] In a possible embodiment of the third aspect, the perceptual information further indicates the voxels of the road surface in the first scene, and the processing unit is configured to specifically generate road surface shape information of the first scene based at least on the perceptual information, and to adjust the suspension in the vehicle based on the road surface shape information.

[0067] In a possible embodiment of the third aspect, the processing unit specifically adjusts the vehicle's driving path based at least on perceptual information, and the adjusted driving path is configured not to pass through areas where obstacle voxels are located.

[0068] In a possible embodiment of the third aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network.

[0069] The image feature extraction network is configured to extract 3D image features from image data.

[0070] The point cloud feature extraction network is configured to extract point cloud features from voxels corresponding to the point cloud data.

[0071] The feature fusion network is configured to perform fusion based on 3D image features and point cloud features of voxels corresponding to point cloud data to obtain the fused features of voxels in the first scene.

[0072] The output network is configured to process the fused voxel features of the first scene and output perceptual information.

[0073] In a possible embodiment of the third aspect, a processing unit inputs text query information and fused features of obstacle voxels into an attribute recognition network and outputs obstacle category information, the text query information is further configured to be used to request a category query, a display unit displays the obstacle category information, the fused features of obstacle voxels are determined based on corner point information of a polygon box corresponding to the obstacle and fused features of voxels in a first scene, the polygon box corresponding to the obstacle is further configured to be associated with the obstacle voxels.

[0074] In a possible embodiment of the third aspect, a receiving unit is further configured to acquire multiple planned routes for a vehicle; a processing unit inputs the multiple planned routes for the vehicle and the fused voxel features of a first scene into a route evaluation network and outputs recommendation coefficients for the multiple planned routes and a recommended route in the multiple planned routes, the recommended route being further configured to be associated with the recommendation coefficients for the multiple planned routes; and a display unit is further configured to display the recommended route.

[0075] According to a fourth aspect, the present application provides an apparatus for intelligent operation. The apparatus includes a processor and memory, the memory being configured to store program instructions, the processor calling program instructions in memory, and as a result the apparatus performs a method according to the first aspect or any one of the possible embodiments of the first aspect.

[0076] According to the fifth aspect, the present application provides a vehicle. The vehicle includes a system according to the second aspect or one of possible embodiments of the second aspect, or an apparatus according to the third aspect or one of possible embodiments of the third aspect, or an apparatus according to the fourth aspect.

[0077] According to a sixth aspect, the present application provides a computer-readable storage medium containing computer instructions. When the computer instructions are executed by a processor, a method according to the first aspect or any possible embodiment thereof is performed.

[0078] According to a seventh aspect, the present application provides a computer program product. When the computer program product is executed by a processor, a method according to the first aspect or one of possible embodiments of the first aspect is performed. For example, the computer program product may be a software installation package. When it is necessary to use a method provided in any possible design of the first aspect, the computer program product may be downloaded and executed on a processor in order to perform a method according to the first aspect or one of possible embodiments of the first aspect. [Brief explanation of the drawing]

[0079] [Figure 1] This is a diagram of a communication system according to one embodiment of the present application. [Figure 2] This is a diagram of a system of a perceptual model for intelligent driving according to one embodiment of the present application. [Figure 3] This is a diagram illustrating feature extraction of a perception detection network according to one embodiment of this application. [Figure 4] This is a flowchart of an intelligent driving method according to one embodiment of this application. [Figure 5] These are diagrams of several scenes according to one embodiment of this application. [Figure 6A]This figure shows how to create an obstacle in a scene using a polygon box according to one embodiment of this application. [Figure 6B] This is a diagram showing the voxels of an obstacle according to one embodiment of this application. [Figure 6C] This figure shows a road surface voxel displayed within a scene according to one embodiment of the present application. [Figure 7A] This is a flowchart of a method for training a perceptual detection network according to one embodiment of this application. [Figure 7B] This is a diagram showing the training process of a perception detection network according to one embodiment of this application. [Figure 8] This is a diagram showing the hardware structure of a chip according to one embodiment of this application. [Figure 9A] This is a diagram showing the structure of a computing device according to one embodiment of the present application. [Figure 9B] This is a diagram showing the structure of a training device according to one embodiment of this application. [Figure 10] This is a diagram showing the structure of a processing device according to one embodiment of this application. [Modes for carrying out the invention]

[0080] It should be noted that the prefixes "first," "second," etc., in this application are merely intended to distinguish different objects and do not impose any restrictions on the arrangement, order, priority, quantity, or content of the described objects. For example, if the object being described is a "field," the ordinal number preceding "field" in "first field" and "second field" does not limit the position or order of the "field." "First" and "second" do not limit whether the "fields" described by "first" and "second" are in the same message, nor do they limit the order of "first field" and "second field." As another example, if the object being described is a "level," the ordinal number preceding "level" in "first level" and "second level" does not limit the priority of the "level." As yet another example, the number of objects being described is not limited by the prefix, and there may be one or more objects being described. "First device" is used as an example, and there may be one or more "devices." Furthermore, objects described by different prefixes may be the same or different. For example, if the object being described is a “device,” then “the first device” and “the second device” may be the same device, the same type of device, or different types of devices. As another example, if the object being described is “information,” then “the first piece of information” and “the second piece of information” may be the same piece of information or different pieces of information. In conclusion, in the embodiments of this application, the use of prefixes used to distinguish the objects being described does not constitute any limitation on the objects being described. For a description of the objects being described, please refer to the claims or the contextual description in the embodiments. The use of prefixes should not constitute an unnecessary limitation.

[0081] In embodiments of this application, for example, the descriptive phrase "at least one of a1, a2, ..., and an" is used and includes cases where any one of a1, a2, ..., and an exists independently, and also includes cases where any combination of a1, a2, ..., and an exists, and it should be noted that each case may exist independently. For example, the descriptive phrase "at least one of a, b, and c" includes cases where there is a single a, a single b, a single c, a combination of a and b, a combination of a and c, a combination of b and c, or a combination of a, b, and c.

[0082] To facilitate understanding, the following will first explain the relevant terms that may be used in the embodiments of this application.

[0083] (1) Autonomous driving Autonomous driving may also be called intelligent driving or assisted driving, and is a key direction in the development of vehicle intelligence. With advancements in perception technology and improvements in chip performance, intelligent driving will provide people with increasingly richer driving capabilities and gradually realize different levels of driving experience. The Society of Automotive Engineers (SAE) provides driving automation standards, including driving levels L0 to L5. Level L0 is no automation. The human driver is fully responsible for operating the vehicle and may receive warnings or assistance from the driving system during driving, such as autonomous emergency braking (AEB), blind spot monitoring (BSM), or lane departure warning (LDW). Level L1 is driver assistance. Level L2 is partial automation. For more steering and acceleration / deceleration operations based on the driving environment, a vehicle following function is provided in combination with driving support, such as adaptive cruise control (ACC) and lane keeping assistance / support (LKA / LKS), and other driving operations are performed by the human driver. Level L3 is conditional automation. The driving system may perform all driving operations. However, the human driver must respond to the driving system's requests at the appropriate time; that is, the human driver must be ready to take over from the driving system. Level L4 is fully automated. The driving system is capable of performing all driving operations, and the human driver does not necessarily need to respond to the driving system's requests.For example, when road and environmental conditions permit (e.g., on a closed campus, highway, city road, or fixed driving route), a human driver may not need to take over driving. Level L5 is fully automated. The driving system may independently complete driving operations under a variety of road and environmental conditions that can be handled by a human driver. At levels L0-L2, the driving system primarily provides support to the driver, who still needs to manage driving to ensure safety and steer, brake, or accelerate as needed. At levels L3-L5, the driving system can perform all driving operations on behalf of the driver. At level L3, the driver needs to be ready to take over driving. At levels L4 and L5, the driving system can perform full driving under some or all conditions, and the driver can choose whether or not to take over driving.

[0084] The above classification is an example. Based on technological advancements or different regulations in different countries or regions, the aforementioned classification may change. For example, the vehicle automation classification proposed by the Chinese Ministry of Industry and Information Technology includes six levels of vehicle driving automation, where levels 0-2 are driver assistance, where the system assists a human in performing dynamic driving tasks, and the driving subject remains the human; and levels 3-5 are autonomous driving, where the system performs dynamic driving tasks on behalf of a human under designed driving conditions, and when the function is activated, the driving subject is the system. The names and definitions of the levels are as follows: Level 0 driving automation (emergency assistance) systems cannot continuously control the horizontal or vertical motion of the vehicle in dynamic driving tasks, but have the ability to continuously detect and respond to several targets and events in dynamic driving tasks. Level 1 driving automation (partial driver assistance) systems can continuously control the horizontal or vertical motion of the vehicle in dynamic driving tasks under the system's designed driving conditions (called the Operation Design Domain ODD), and have the ability to detect and respond to several targets and events corresponding to the horizontal or vertical motion control of the vehicle performed. Level 2 combined driver assistance systems continuously control the vehicle's horizontal and vertical motion in dynamic driving tasks under the system's designed driving conditions, and have the ability to detect and respond to several targets and events corresponding to the horizontal and vertical motion controls performed. Level 3 conditionally automated driving systems continuously perform all dynamic driving tasks under the system's designed driving conditions. Level 4 highly automated driving systems continuously perform all dynamic driving tasks and automatically perform minimum-risk maneuvers under the system's designed driving conditions. Level 5 fully automated driving systems continuously perform all dynamic driving tasks and automatically perform minimum-risk maneuvers under any driving conditions.Horizontal control is primarily used for vehicle steering control, such as controlling the torque and angle of the steering wheel to control the direction of the vehicle. Vertical control is primarily used for vehicle speed control, such as controlling the brake pedal, accelerator pedal, gears, etc., to control the acceleration, deceleration, and braking of the vehicle.

[0085] Regardless of the classification scheme used, the descriptions in the embodiments of this application are applicable to the aforementioned autonomous driving systems that are required to be fully or partially involved in the driving of the vehicle.

[0086] (2) Obstacles In embodiments of this application, an obstacle is an entity that is not expected to collide with a vehicle during the driving process. The entity may be static or dynamic. A static entity may be a static object having volume and mass, such as a container on the road, a road construction sign, a road boundary rail, a pile of soil, a tire, an overturned vehicle, a person or animal lying down, a roadside building, a tree, a parked vehicle, a road sign, a utility pole, or a roadside curb belt. A dynamic entity may be a moving object having volume and mass, such as a pedestrian (such as a walking pedestrian or cyclist), an animal, a vehicle, or a vehicle carrying goods (e.g., a container, a tree branch, or other goods).

[0087] (3) Scene A scene is the environmental space that can be detected by sensors on the vehicle during the vehicle's driving process. In the vehicle's driving process, each time point corresponds to one scene, and a scene corresponding to multiple times points contains scenes corresponding to each of those times.

[0088] (4) Voxel A voxel is sometimes called a three-dimensional pixel or volume element. A voxel is the smallest unit for separation in three-dimensional space, similar to a pixel in two-dimensional space. The three-dimensional space may be divided into a grid, and features may be assigned to each grid by using voxels. In this case, voxels represent values ​​on a regular grid in three-dimensional space, and the positioning of voxels may be inferred based on the placement of voxels relative to other voxels.

[0089] The following describes the technical solutions of the embodiments in this application with reference to the attached drawings.

[0090] Figure 1 is a diagram of a communication system according to one embodiment of the present application. As shown in Figure 1, the system includes a network-side device and a vehicle. The network-side device communicates with the vehicle wirelessly.

[0091] Here, the network-side device is a device having computing capabilities. The network-side device may be, for example, a server deployed on the network side (e.g., a server for intelligent operation processing), or a component or chip within a server. In some possible embodiments, the network-side device may also be a system-level device comprising a cluster of multiple servers or computing devices. The network-side device may be deployed in a cloud environment or an edge environment. This is not specifically limited to this embodiment of the present application.

[0092] Here, the vehicle is a vehicle configured with an autonomous driving system. The autonomous driving system is not limited to fully autonomous driving systems, highly autonomous driving systems, conditionally autonomous driving systems, or partially autonomous driving systems. Those skilled in the art will understand that all non-fully manual systems for intelligent driving may be included within this concept.

[0093] For example, depending on the different power sources of the vehicle, the vehicle may be, for example, a new energy vehicle or a conventional vehicle. A conventional vehicle is a fuel-powered vehicle, and may be, for example, a gasoline vehicle or a diesel vehicle. A new energy vehicle may be, for example, an electric vehicle (EV), a hybrid electric vehicle (HEV), a range-extended electric vehicle (EV), a plug-in hybrid electric vehicle (HEV), a fuel cell vehicle, or another new energy vehicle. This is not particularly limited here.

[0094] Cameras and radar are deployed on the vehicle. The cameras are configured to collect image data of the vehicle's current surroundings, and the radar is configured to collect point cloud data of the vehicle's current surroundings. The radar includes at least one of Lidar and millimeter-wave radar. Based on the installation arrangement of the cameras on the vehicle, the cameras may be classified, for example, as front-view cameras, ring-view cameras, rear-view cameras, and side-view cameras. Based on the camera structure, the cameras may be classified, for example, as monocular cameras, binocular cameras, and wide-angle cameras. Here, the number of cameras configured on the vehicle is not limited to this embodiment of the present application. For safety purposes, the cameras on the vehicle need to be able to collect 360-degree image data around the vehicle's body.

[0095] For example, a perception model is deployed on a network-side device, and the network-side device trains the perception model using training data. The training data includes sensor data acquired from a data source device (e.g., a collection vehicle platoon), and the sensor data includes image data collected by an on-board camera and point cloud data collected by an on-board radar. After training the perception model, the network-side device may provide the trained perception model to the vehicle for use. For specific training processes of the perception model, please refer to the corresponding descriptions in the embodiments of the method below. Further details will not be provided again.

[0096] Furthermore, the vehicle may acquire a perceptual model (i.e., a trained perceptual model) from a network-side device. During the vehicle's driving process, the vehicle collects data on the environment (or scene) within a specific range from the vehicle by using sensors mounted on the vehicle (e.g., cameras or radar), and acquires collected data. The collected data includes, for example, image data and point cloud data collected about the scene. The vehicle processes the collected data by using a perceptual model to output perceptual information about the scene. The perceptual information indicates voxels of obstacles in the scene. The vehicle may control its driving based at least on the perceptual information.

[0097] For details of the perceptual model, please refer to the relevant descriptions of the embodiments below in Figure 2. Further details will not be explained again.

[0098] In the system shown in Figure 1, communication between the network-side device and the vehicle may use cellular communication technologies, such as 2G cellular communication including the global system for mobile communication (GSM) or general packet radio service (GPRS), or 3G cellular communication including wideband code division multiple access (WCDMA®), time division-synchronous code division multiple access (TS-SCDMA), or code division multiple access (CDMA), or 4G cellular communication including long-term evolution (LTE) or LTE-based vehicle to everything (V2X), or 5G cellular communication including PC5 communication or new radio (NR)-V2X PC5 communication, or other advanced cellular communication technologies. The wireless communication system may also communicate with a wireless local area network (WLAN) by using non-cellular communication technologies such as Wi-Fi. This is not particularly limited here.

[0099] Figure 1 is merely one example of an architecture diagram, and it should be understood that the number of network elements included in the system shown in Figure 1 is not limited. In addition to the functional entities shown in Figure 1, other functional entities not shown in Figure 1 may be included. Furthermore, the methods provided in the embodiments of this application may be applied to the communication system shown in Figure 1. Indeed, the methods provided in the embodiments of this application may be applied to other communication systems instead. This is not limited to the embodiments of this application.

[0100] Figure 2 is a diagram of a perception model system for intelligent driving according to one embodiment of the present application.

[0101] In Figure 2, the perception model includes a perception detection network. The perception detection network is configured to output perception information based on collected data (e.g., image data and point cloud data) gathered by sensing the scene. The perception information indicates voxels of obstacles in the scene. In some possible embodiments, the perception information further indicates voxels of the road surface in the scene. The perception information may be used to assist in driving a vehicle.

[0102] The following describes the framework of the perception detection network.

[0103] In one embodiment, when the collected data includes image data and point cloud data, the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. The image feature extraction network is configured to extract 3D image features from the image data and output the features to the feature fusion network. The point cloud feature extraction network is configured to extract point cloud features of voxels corresponding to the point cloud data from the point cloud data and output the features to the feature fusion network. The feature fusion network is configured to fuse the 3D image features of the image data and the point cloud features of voxels corresponding to the point cloud data to obtain fused features of voxels in the corresponding scene and output the features to the output network. The output network makes predictions based on the fused features of voxels in the corresponding scene and outputs perception information of the scene.

[0104] For example, the feature fusion network in Figure 2 may perform only spatial feature fusion. For instance, the image data is an image captured by a camera at time t, and the point cloud data is data collected by radar at time t. In this case, the feature fusion network only needs to perform spatial fusion on the 3D image features of the image data at time t and the point cloud features of the voxels corresponding to the point cloud data at time t.

[0105] For example, the feature fusion network in Figure 2 may perform spatial and temporal feature fusion. For instance, image data is image data collected by a camera at n time points, and point cloud data is point cloud data collected by radar at n time points. In this case, the feature fusion network may first perform spatial fusion on the 3D image features and the point cloud features of the voxels corresponding to each of the n time points in order to obtain the spatial fusion features of the voxels corresponding to the time points, and then perform temporal fusion on the spatial fusion features of the voxels corresponding to each of the n time points.

[0106] In some possible embodiments, if the collected data consists only of image data, the point cloud feature extraction module in the perception detection network shown in Figure 2 may be omitted. If the image data is image data collected by a camera at n time points, the feature fusion network performs temporal fusion on the 3D image features corresponding to each of the n time points.

[0107] In some possible embodiments, if the collected data consists only of point cloud data, the image feature extraction module in the perception detection network shown in Figure 2 may be omitted. If the point cloud data is point cloud data collected by radar at n time points, the feature fusion network performs temporal fusion on the point cloud features of the voxels corresponding to each of the n time points.

[0108] For example, a feature fusion network may use a recurrent neural network (RNN) or a recurrent convolutional neural network (RCNN) network structure, and the CNN may be, for example, a long short-term memory network (LSTM) network or a gated recurrent unit network (GRU) network.

[0109] Furthermore, the image feature extraction network includes a camera backbone network and a stereo transformation network, where the camera backbone network is configured to extract 2D image features from image data, and the stereo transformation network is configured to transform the 2D image features from image data into 3D image features from image data. Here, the stereo transformation network may transform the 2D image features into 3D image features in the vehicle coordinate system, or the features extracted from radar point cloud data are 3D features in the vehicle coordinate system. This helps the feature fusion network subsequently perform feature fusion on features from different sensors, helping to eliminate heterogeneous differences between multimodal sensors.

[0110] For example, 2D image features of image data include, but are not limited to, color features, shape features, texture features, and spatial relationship features of the image data.

[0111] For example, the camera backbone network may use a network structure such as a convolutional neural network (CNN) (e.g., a residual network Resnet), a transformer network, a vision transformer (ViT) network, or another backbone network. The stereo conversion network may use a network structure such as a transformer network or a lift-splat-shoot (LSS) network.

[0112] Furthermore, the point cloud feature extraction network includes a radar coding network and a point backbone network. The radar coding network is configured to voxelize the point cloud data, establish correspondences between points and voxels in the point cloud data, and obtain features of the voxels corresponding to the point cloud data. The point backbone network is configured to extract point cloud features (i.e., 3D features) of the voxels corresponding to the point cloud data based on the features of the voxels corresponding to the point cloud data. In some possible embodiments, the radar coding network and the point backbone network may be combined into a single network to extract point cloud features of the voxels corresponding to the point cloud data. This is not particularly limited here. In some possible embodiments, when computing power is supported, the radar coding network and the point backbone network may also be combined into a single network, and this network may extract point cloud features of the voxels corresponding to the point cloud data.

[0113] For example, a radar coding network may use a voxel feature encoding (VFE) network or a pillar feature encoding (PFE) network structure. A point backbone network may use a convolutional neural network (e.g., U-Net) or a transformer network structure.

[0114] The output network is the detection head of the perception detection network. The output network includes at least one head network, and the number of head networks in the output network is determined based on the number of types of prediction results in the perception information output by the output network. For example, as shown in Figure 2, the perception information includes voxel occupancy status, voxel velocity information, voxel visibility status, and corner point information of polygon boxes corresponding to obstacles. The polygon boxes corresponding to obstacles are associated with the obstacle voxels. Thus, it can be seen that the perception information includes four types of prediction results. Therefore, the output network includes four head networks: head network 1, head network 2, head network 3, and head network 4. Head network 1 is configured to output corner point information of polygon boxes corresponding to obstacles, head network 2 is configured to output voxel occupancy status, head network 3 is configured to output voxel velocity information, and head network 4 is configured to output voxel visibility status.

[0115] Here, the visibility status of a voxel means that, in a scene in which the vehicle is currently positioned, if a particular voxel in that scene is not detected by any observational signals from any sensors on the vehicle (including cameras and radar) at the current time, the visibility status of that voxel is invisible, and if the voxel is detected by an observational signal from at least one sensor, the visibility status of the voxel is visible.

[0116] For example, the occupation status of a voxel is such that, in a scene where a vehicle is currently positioned, if an entity exists in the spatial location in the physical world where the scene is positioned, the occupation status of this voxel is occupied; if no entity exists in the spatial location in the physical world where the scene is positioned, the occupation status of this voxel is empty (i.e., unoccupied). Here, an entity may be understood as an object with a specific volume and mass. It will be understood that air is not an entity.

[0117] For example, any head network within the output network may use the network structure of a convolutional neural network (CNN) or a transformer network. Here, the internal network structures of different head networks within the output network may be identical or different. It will be understood that different head networks process the same input features in different ways.

[0118] In some possible embodiments, to reduce computing power consumption, a neural sampling network may be further placed between the feature fusion network and the output network in the perception detection network shown in Figure 2. Specifically, the feature fusion network outputs the fused features of the scene's voxels to the neural sampling network, which processes the fused features of the scene's voxels by using different resolutions based on the importance of the regions where the scene's voxels are located. For example, if the importance of region 1 in the scene is greater than that of region 2 in the scene, the fused features of the voxels in region 1 are processed using a first resolution, and the fused features of the voxels in region 2 are processed using a second resolution, where the first resolution is greater than the second resolution. In this way, the neural sampling network can perform fine-grained processing on voxels in important regions of the scene and coarse-grained processing on voxels in less important regions of the scene. This can significantly reduce computing power, help improve the data processing efficiency of the perception detection network, and also help reduce hardware deployment costs.

[0119] For example, when both Region 1 and Region 2 satisfy at least one of the following conditions, the importance of Region 1 is greater than the importance of Region 2. (1) The distance between region 1 and the vehicle is less than the distance between region 2 and the vehicle. (2) The number of obstacles in area 1 is greater than the number of obstacles in area 2. (3) The number of dynamic obstacles in region 1 is greater than the number of dynamic obstacles in region 2. (4) The volume of the obstacle in region 1 is greater than the volume of the obstacle in region 2.

[0120] For example, a neural sampling network may use a neural network, a multi-layer perceptron (MLP), or a transformer network.

[0121] Refer to Figure 3 for a clearer illustration of the feature extraction procedure of the perception detection network. Figure 3 is a diagram of feature extraction of a perception detection network according to one embodiment of the present application. In Figure 3, 2D image features of image data may be extracted using a camera backbone network based on image data collected by n cameras. 3D image features of image data may be extracted using a stereo transformation network based on the 2D image features of image data. Voxel features (i.e., 3D features) corresponding to point cloud data may be extracted using a radar coding network based on point cloud data collected by radar. Point cloud features (i.e., 3D features) of voxels corresponding to point cloud data may be extracted using a point backbone network based on the voxel features corresponding to point cloud data. The 3D image features of image data and the point cloud features of voxels corresponding to point cloud data are fused using a feature fusion network to output fused voxel features. Finally, based on the voxel fusion features, the output network outputs the voxel occupancy status, voxel velocity information, voxel visibility status, and polygon box corner point information corresponding to obstacles.

[0122] Figure 3 is merely one example of the feature extraction process in a perceptual detection network, and it should be understood that the feature extraction process in a perceptual detection network is not limited to what is shown in Figure 3.

[0123] In some possible embodiments, the perception model further includes an attribute recognition network. The attribute recognition network may be configured to recognize the category of an obstacle. For example, the attribute recognition network may be configured to output obstacle category information based on text query information and fused features of the obstacle's voxels, where the text query information is used to request a category query, the fused features of the obstacle's voxels are determined based on corner point information of the polygon box corresponding to the obstacle and fused features of the scene's voxels, where the polygon box corresponding to the obstacle is associated with the obstacle's voxels. From Figure 2, it can be seen that the corner point information of the polygon box corresponding to the obstacle comes from the output network within the perception detection network (specifically, head network 1 within the output network), and the fused features of the scene's voxels are the output of the feature fusion network within the perception detection network.

[0124] Here, the association of a polygon box corresponding to an obstacle with the voxel of the obstacle may be understood as the acquisition of corner point information for the polygon box corresponding to the obstacle based on the index information of the voxel of the obstacle. For example, the corner point information for the polygon box corresponding to an obstacle may be acquired by the head network 1 in Figure 2 by predicting the index information of the voxel of the obstacle based on learned rules, or it may be acquired computationally by using a convex hull algorithm based on the index information of the voxel of the obstacle, and is not limited thereto.

[0125] Here, the polygon boxes corresponding to obstacles may be two-dimensional or three-dimensional; this is not particularly limited here.

[0126] In one embodiment, the attribute recognition network includes a text encoding network and an attribute decoding network, the text encoding network being configured to extract word vector features from text query information, and the attribute decoding network being configured to output obstacle category information based on the word vector features and the fused features of the obstacle voxels.

[0127] Here, text query information is used to request a query for Q categories. The number of categories of obstacles in a scene is P, and we assume that Q and P are positive integers, and that Q is greater than P. That is, the number of categories actually supported by the attribute recognition network for recognition is greater than the number of categories of obstacles in any scene. In this way, it can be guaranteed that omissions can be avoided when the attribute recognition network performs category recognition for obstacles in any scene.

[0128] For example, in a category inference process, the text query information includes K text query statements, such as "Is it a vehicle?", "Is it a pedestrian?", "Is it a utility pole?", "Is it a road sign?", "Is it a road boundary rail?", etc. The text coding network within the attribute recognition network performs feature extraction on the K text query statements and obtains word vector features corresponding to each text query statement. The word vector features corresponding to each text query statement may represent the image semantic features of the category indicated by the text query statement. An example is used in which the attribute decoding network within the attribute recognition network recognizes the category of obstacle 1. Obstacle 1 is any obstacle in the scene. The attribute decoding network calculates similarity between the fused voxel features of obstacle 1 and the word vector features corresponding to each of the K text query statements, and determines that the category corresponding to the word vector feature with the highest similarity to the fused voxel features of obstacle 1 is the category of obstacle 1, and as a result, the category information of obstacle 1 may be output.

[0129] Here, determining the voxel fusion features of obstacle 1 based on the corner point information of the polygon box corresponding to obstacle 1 and the voxel fusion features of the scene may mean that the corner point information of the polygon box corresponding to obstacle 1 corresponds to the voxel index information of obstacle 1, and therefore, the voxel fusion features of obstacle 1 may be determined from the voxel fusion features of the scene based on the voxel index information of obstacle 1.

[0130] For example, both the text encoding network and the attribute decoding network may use the network structure of a convolutional neural network or a transformer network. It will be understood that the text encoding network and the attribute decoding network may adaptively adjust the relevant network parameters based on the functions of the text encoding network and the attribute decoding network.

[0131] In some possible embodiments, the perception model may further include a route evaluation network, which may be configured to determine a recommended route for the vehicle. For example, the route evaluation network may be configured to output recommendation coefficients for multiple planned routes and recommended routes for multiple planned routes, based on multiple planned routes for the vehicle and fused voxel features of the scene.

[0132] In one embodiment, the route evaluation network includes a route coding network, a feature interaction network, and an evaluation output network. The route coding network is configured to extract the route features of each of several planned routes for a vehicle. The feature interaction network is configured to obtain the risk features of each planned route based on the route features of each planned route and the fused features of the scene voxels. The evaluation output network is configured to output recommendation coefficients for the multiple planned routes and the recommended route for the multiple planned routes based on the risk features of the multiple planned routes.

[0133] For example, the recommendation factor for a planned route may be obtained based on at least one of the following: the risk factor of the planned route, comfort, and traffic efficiency. The risk factor of the planned route is related to at least one of the following factors: the distance between the planned route and obstacles (including visible obstacles and obstacles currently in blind spots), and whether the planned route collides with the routes of other traffic participants on the road (e.g., whether a collision occurs at the present or future time). The traffic efficiency of the planned route is related to at least one of the following factors: the length of the planned route, the estimated traffic duration corresponding to the planned route, the number of traffic lights on the planned route, and the area of ​​the drivable region on which the planned route is located. The comfort of the planned route is related to at least one of the following factors: the values ​​of the steering acceleration and steering frequency of the planned route, the rate of change of the acceleration of the planned route, the flatness of the road surface of the planned route, the number of traffic lights on the planned route, the type of road on which the planned route is located, and whether the route area of ​​the planned route is cold.

[0134] For example, the recommended route is the route that corresponds to the highest recommendation coefficient among multiple planned routes.

[0135] For example, the path coding network may use a convolutional neural network, transformer network, graph neural network (GNN), or graph convolutional neural network (GCNN) network structure. The feature interaction network may use a graph neural network or transformer network network structure. The evaluation output network may use a neural network or a multilayer perceptron MLP network structure.

[0136] It should be understood that the perceptual model framework shown in Figure 2 is merely a feasible example provided in the embodiments of this application and should not constitute a limitation on the perceptual model framework.

[0137] For example, when a perception model includes a perception detection network, an attribute recognition network, and a path evaluation network, the training of the perception detection network, attribute recognition network, and path evaluation network may be separated. For instance, the perception detection network may be trained first, followed by the attribute recognition network and path evaluation network. Alternatively, the training of the perception detection network, attribute recognition network, and path evaluation network may be performed simultaneously. This is not particularly limited here. For the training process of each network in the perception model, please refer to the corresponding descriptions in the embodiments below. Further details will not be explained again.

[0138] Figure 4 is a flowchart of an intelligent driving method according to one embodiment of the present application. The method may be applied to a vehicle or components (e.g., chips or integrated circuits) used for automated driving control on a vehicle, as shown in Figure 1, or at least a perception detection network deployed on the vehicle. The method includes, but is not limited to, the following steps:

[0139] S401: Sensor data is collected for the first scene, and the sensors include at least one of a camera and a radar.

[0140] Here, the first scene may be understood as the environmental space that can be detected by sensors during the vehicle's driving process.

[0141] Here, the sensors are deployed on the vehicle. Based on the camera placement on the vehicle, the cameras may be classified, for example, as front-view cameras, ring-view cameras, rear-view cameras, and side-view cameras. The radar includes at least one of lidar and millimeter-wave radar. The number of cameras and radars configured on the vehicle is not limited to this embodiment of the present application.

[0142] The camera is configured to collect image data, and the radar is configured to collect point cloud data. Therefore, the collected data includes at least one of the image data and point cloud data.

[0143] For example, multiple cameras may be configured on a vehicle. Different cameras may have different fields of view, and as a result, the fields of view of the multiple cameras may cover a 360-degree field of view centered on the vehicle. For example, the fields of view of adjacent cameras among the multiple cameras may overlap. In this way, data within the same environmental space may be collected simultaneously by multiple sensors, which helps to improve the reliability of data observation.

[0144] For example, sensors include cameras and radar. Assume there are m cameras on the vehicle. The m cameras collect image data for a first scene. Assume each camera captures one image at each time point, i.e., the collected data corresponding to each time point includes image data corresponding to the m images collected by the cameras and point cloud data collected by the radar.

[0145] S402: The collected data is input into the perception detection network, and perception information is output, which indicates the voxels of obstacles in the first scene.

[0146] Here, an obstacle is an entity that is not expected to collide with a vehicle during the driving process. Entities may be static or dynamic. Static entities may be static objects with volume and mass, such as containers on the road, road construction signs, road boundary rails, soil deposits, tires, overturned vehicles, lying people or animals, roadside buildings, trees, parked vehicles, road signs, utility poles, or roadside curbing belts. Dynamic entities may be moving objects with volume and mass, such as pedestrians (such as walking pedestrians or cyclists), animals, vehicles, or vehicles carrying goods (e.g., containers, tree branches, or other items).

[0147] Here, the perception detection network is a trained and vehicle-side perception detection network. The perception detection network is configured to output perceptual information based on sensor data for a first scene. For example, the perception detection network is acquired by the network-side device shown in Figure 1 by training it on a sensor dataset and corresponding label information generated by 4D reconstruction. The label information corresponding to the sensor dataset may be generated by the network-side device by performing a self-supervised 4D reconstruction based on the sensor dataset. The label information is used in the training process of the perception detection network to provide actual value information for the perception detection network's prediction results.

[0148] For example, the prediction task of a perception detection network includes four prediction tasks: predicting the occupancy status of a voxel, the velocity information of a voxel, the visibility status of a voxel, and the corner point information of a polygon box corresponding to an obstacle. In the training process of the perception detection network, it is assumed that the current input data of the perception detection network is image data and point cloud data at time t. In this case, the perception detection network processes the input data using the four prediction tasks described above and outputs predicted perception information (i.e., prediction results). Correspondingly, the label information includes actual value information of the prediction results corresponding to the image data and point cloud data at time t.

[0149] In one embodiment, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. In this case, for example, please refer to steps A1 to A4 below for the processing steps of the perception detection network.

[0150] A1: The image feature extraction network extracts 3D image features from image data.

[0151] A2: The point cloud feature extraction network extracts point cloud features from voxels corresponding to the point cloud data.

[0152] A3: The feature fusion network fuses 3D image features with point cloud features of voxels corresponding to point cloud data in order to obtain fused features of voxels in the first scene.

[0153] A4: The output network processes the fused voxel features of the first scene and outputs perceptual information.

[0154] For a detailed explanation of the perception detection network's inference process, please refer to the description of the perception detection network in the embodiment shown in Figure 2. For the image feature extraction network, point cloud feature extraction network, feature fusion network, and output network, please refer to the corresponding descriptions in the embodiment shown in Figure 2. Further details will not be provided. It will be understood that the above example does not constitute a limitation on the framework of the perception detection network.

[0155] In this embodiment of the present application, the perceptual information includes at least one of the following information: the occupancy status of a voxel in a first scene, velocity information of a voxel in a first scene, visibility status of a voxel in a first scene, and corner point information of a polygon box corresponding to an obstacle in a first scene, wherein the polygon box corresponding to an obstacle in a first scene is associated with the voxel of the obstacle.

[0156] For example, see the perception detection network framework shown in Figure 2. The output network includes four head networks, each corresponding to one prediction task. In this case, the perception information includes the occupancy status of voxels in the first scene, velocity information of voxels in the first scene, visibility status of voxels in the first scene, and corner point information of polygon boxes corresponding to obstacles in the first scene.

[0157] Here, the occupation status of voxels may be classified into two types: "occupied" and "empty." For more information on voxel occupation status, please refer to the related explanation above. Further details will not be provided again.

[0158] For example, if voxel 1 of the scene corresponds to vehicle A in the physical world where the scene is located, the occupation status of voxel 1 is "occupied," and if voxel 2 of the scene corresponds to air in the physical world where the scene is located, the occupation status of voxel 2 is "empty."

[0159] Here, the visibility status of a voxel may have two types: "visible" and "invisible." For more information on voxel visibility status, please refer to the related explanation above. Further details will not be provided again.

[0160] For example, the visibility status of a voxel may be changed. Figure 5 shows several scenes according to one embodiment of the present application. Figure 5(1) shows Scene 1 corresponding to time t1. In Figure 5(1), Vehicle 1 is the vehicle itself (specifically, the perception detection network is deployed on Vehicle 1). Vehicles 1, 2, and 3 are positioned in the same lane, and it can be seen that Vehicle 2 is currently performing a lane change operation. It is assumed that the body shape of Vehicle 2 is larger than the body shape of Vehicle 3 in front. As a result, Vehicle 3 is completely obscured by Vehicle 2 from the perspective of Vehicle 1. Vehicle 3 is positioned in the blind spot of Vehicle 1. Therefore, observation signals from any sensor on Vehicle 1 at time t1 cannot detect the voxel of Vehicle 3. In this case, in the perception information output by Vehicle 1, the visibility status of the voxel of Vehicle 2 at time t1 is "visible", but the visibility status of the voxel of Vehicle 3 at time t1 is "invisible". Figure 5(2) shows Scene 2 corresponding to time t2. It can be seen that Vehicle 2 has just completed a lane change operation. It is assumed that both Vehicle 2 and Vehicle 3 are in the field of view acquired by the sensors of Vehicle 1. In other words, this means that at time t2, both the voxels of Vehicle 2 and Vehicle 3 can be detected by the observation signal of at least one sensor on Vehicle 1. Therefore, in the perceptual information output by Vehicle 1, the visibility status of the voxels of Vehicle 2 at time t2 is "visible," and the visibility status of the voxels of Vehicle 3 at time t2 is also "visible." Thus, it can be seen that data collected at multiple times is input to the perception detection network, and the perception detection network not only supplements the observation information but also reconstructs the physical world in which the scene is set more realistically from multiple viewpoints.

[0161] S403: Control the operation of the vehicle based at least on perceptual information.

[0162] Here, controlling the operation of the vehicle includes at least one of the following: lane changes, driving speed adjustment, driving path adjustment, warning light activation, and vehicle suspension adjustment. In this way, the vehicle facilitates real-time decision-making based on at least perceptual information to improve safety in the vehicle driving process.

[0163] In one embodiment, the step of controlling the driving of a vehicle based on at least perceptual information includes the step of adjusting the driving path of the vehicle based on at least perceptual information such that the adjusted driving path does not pass through areas where obstacle voxels are located.

[0164] For example, based on perceptual information, it may be determined whether the vehicle's current driving path will collide with an obstacle voxel at the corresponding time in the scene at the current time and future time. When a collision is predicted to occur, the vehicle's current driving path may be adjusted in a timely manner, so that the adjusted driving path does not pass through the area where the obstacle voxel is located. In this way, collisions between the vehicle and obstacles in the driving process can be avoided, thereby improving the driving safety of the vehicle.

[0165] In one embodiment, the perceptual information further indicates the voxels of the road surface in a first scene, and the steps of controlling the driving of the vehicle based on the perceptual information include at least the steps of generating road surface shape information of the first scene based on the perceptual information and adjusting the suspension in the vehicle based on the road surface shape information.

[0166] Here, road surface shape information indicates the status of the road surface in the first scene (for example, whether the road surface has dips or bumps). The vehicle may acquire the status of the road surface ahead of the vehicle in advance based on perceptual information. When it is detected that the road surface is changing, the vehicle has enough time to adjust its suspension in a timely manner so that the vehicle can maintain a level and stable state as much as possible during the driving process, reduce vibrations caused by road surface changes, and improve the ride comfort of the vehicle.

[0167] For example, controlling vehicle operation based on perceptual information could alternatively involve determining information about blind spots and obstacles within the scene (e.g., obstacle speed information, corner point information of polygon boxes corresponding to obstacles) based on perceptual information, and then controlling the vehicle to decelerate, stop, or steer when it approaches a blind spot, based on the information about obstacles within the blind spot. Here, the category and presentation form of obstacles within the blind spot are not limited, and obstacles within the blind spot may be static or dynamic. This is not particularly limited here. In this way, when the vehicle approaches a blind spot in the scene at the current time, the vehicle is controlled to decelerate, stop, or steer, thus avoiding collisions between the vehicle and obstacles within the blind spot and improving vehicle driving safety.

[0168] Here, a blind spot is, for example, an area where voxels whose visibility status at the current time is "invisible" in perceptual information are located. For example, a blind spot includes the detection blind spot of a sensor and an area that can be detected by the observation signal of a sensor on the vehicle at the current time but cannot be detected due to occlusion by another obstacle.

[0169] Here, the velocity information of the obstacle may be obtained, for example, based on the velocity information of the obstacle's voxels.

[0170] In some possible embodiments, in addition to controlling the driving of the vehicle based on perceptual information output by the vehicle, the vehicle may further control the driving of the vehicle based on at least one of the following: navigation map information, high-resolution map information, live traffic information broadcast by a roadside device, live traffic information broadcast by another surrounding vehicle, etc. Here, the roadside device may be a device, for example, a roadside unit (RSU), multi-access edge computing (MEC), or a sensor, or an internal component or chip of the device, or a system-level device including an RSU and an MEC, or a system-level device including an RSU and a sensor, or a system-level device including an RSU, an MEC, and a sensor.

[0171] Optionally, in some possible embodiments, the intelligent driving method further includes the steps of displaying obstacles in a first scene based on perceptual information, wherein the obstacles in the first scene are marked by using polygon boxes, and / or displaying voxels of the obstacles in the first scene based on perceptual information.

[0172] For example, an obstacle or a voxel of an obstacle may be presented on the vehicle's display device. For example, the display device may be an in-vehicle infotainment tablet, an in-vehicle display, a head-up display (HUD) system, an augmented reality (AR) head-up display (HUD) system, etc. This is not particularly limited herein.

[0173] Figure 6A illustrates the creation of obstacles in a scene using polygon boxes according to one embodiment of the present application. Figure 6A shows obstacles in a scene where the host vehicle is currently positioned, and the obstacles are marked using polygon boxes. In Figure 6A, the vehicle below the center is the host vehicle. It can be seen that obstacles around the host vehicle in the scene are marked and displayed using polygon boxes. Based on the shape of the polygon boxes, it can be seen that the obstacles in the scene include at least vehicles, buildings, etc. For example, the polygon boxes may be two-dimensional or three-dimensional. In Figure 6A, the polygon box closest to the host vehicle on its right side is used as an example. When the polygon boxes are displayed in 2D mode, they may be formed by connecting 10 corner points indicated by one set of corner point information. When the polygon boxes are displayed in 3D mode, they may be formed by connecting corner points indicated by multiple sets of corner point information, each set of corner point information indicating 10 corner points. In some possible embodiments, for dynamic obstacles in the scene, arrows may be further added to the polygon boxes corresponding to the dynamic obstacles. The arrows indicate that the obstacle is a dynamic obstacle, the direction of the arrow indicates the direction of the obstacle's movement, and the length of the arrow indicates the obstacle's speed. Figure 6A is merely an example of displaying the marking of obstacles in a scene where a vehicle is positioned at a particular time, and it will be understood that this does not constitute a limitation on displaying the marking of obstacles in a scene where a vehicle is positioned.

[0174] Figure 6B is a diagram showing an obstacle voxel according to one embodiment of the present application. Figure 6B shows an obstacle voxel in a scene in which the host vehicle is located at the present time. It can be seen that the obstacle voxel comprises multiple voxels in the scene, and that a voxel can be understood as the smallest unit cube in Figure 6B. For example, in Figure 6B, dynamic obstacles and static obstacles may be distinguished and displayed with different colors (in other words, obstacles of different speeds may be distinguished by using different colors), or different obstacles may be distinguished by using different colors. This is not particularly limited here. It will be understood that Figure 6B is merely an example of displaying an obstacle voxel in a scene in which a vehicle is located at a particular time, and does not constitute a limitation on displaying an obstacle voxel in a scene in which a vehicle is located.

[0175] In some possible embodiments, the voxels of the road surface in the scene in which the vehicle is currently located may also be displayed. Figure 6C is a diagram showing the voxels of the road surface in a scene according to one embodiment of the present application. In Figure 6C, not only the voxels of obstacles in the scene at the current time, but also the voxels of the road surface in the scene at the current time are displayed. In this way, the degree of variation of the road surface ahead can be seen based on Figure 6C. It will be understood that Figure 6C is merely an example of displaying the voxels of obstacles and the road surface in a scene in which the vehicle is located at a particular time, and does not constitute a limitation on displaying the voxels of obstacles and the road surface in a scene in which the vehicle is located.

[0176] In some possible embodiments, in addition to the perception detection network, an attribute recognition network may be further deployed on the vehicle, and the attribute recognition network may be configured to recognize the category of obstacles. In this way, the vehicle can not only detect obstacles in its surroundings but also recognize the category of obstacles in the driving process, thereby enabling the vehicle to not only see objects but also understand them.

[0177] Furthermore, the intelligent driving method further includes the steps of acquiring text query information, inputting the text query information and the fused voxel features of the obstacle into an attribute recognition network and outputting category information of the obstacle, wherein the text query information is used to request a category query, and displaying the category information of the obstacle. The fused voxel features of the obstacle are determined based on the corner point information of the polygon box corresponding to the obstacle and the fused voxel features of the first scene, and the polygon box corresponding to the obstacle is associated with the obstacle voxels. Both the corner point information of the polygon box corresponding to the obstacle and the fused voxel features of the first scene are from the perception detection network. Furthermore, based on the perception detection network shown in Figure 2, it can be seen that the corner point information of the polygon box corresponding to the obstacle is from the output network within the perception detection network, and the fused voxel features of the first scene are from the feature fusion network within the perception detection network. For details of this embodiment, please refer to the relevant description of the attribute recognition network in the embodiment in Figure 2. For brevity, further details will not be described here.

[0178] In some possible embodiments, after perceptual information is acquired, further fusion processing may be performed based on the detection results of a detection algorithm configured in the camera, a detection algorithm configured in the radar, or a detection result of another model. In this way, the reliability of obstacle detection is higher when the same obstacle can be detected in multiple different ways.

[0179] In some possible embodiments, in addition to the perception detection network, a route evaluation network may be further deployed on the vehicle, and the route evaluation network is used to recommend the lowest-risk route to the vehicle. This helps improve driving safety and the accuracy of driving decisions.

[0180] Furthermore, the intelligent driving method includes the steps of: acquiring multiple planned routes for a vehicle; inputting the multiple planned routes for the vehicle and the fused voxel features of a first scene into a route evaluation network and outputting recommendation coefficients for the multiple planned routes and recommended routes in the multiple planned routes, wherein the recommended routes are associated with the recommendation coefficients for the multiple planned routes; and displaying the recommended routes. It can be seen that the fused voxel features of the first scene are from a perception detection network. Based on the description of the perception detection network in Figure 2, it can be seen that the fused voxel features of the first scene are provided by a feature fusion network within the perception detection network. Here, the multiple planned routes are generated by the vehicle. For example, the vehicle generates multiple planned routes based on navigation map information. For details of this embodiment, please refer to the relevant description of the route evaluation network in the embodiment in Figure 2. For the sake of brevity of the specification, further details will not be described again.

[0181] For example, the recommended route output by the route evaluation network includes at least two of several planned routes. In this case, in a human-computer collaborative driving scenario, the recommended route may be further used by the user to whom it is recommended, and feedback information is received from the user, indicating the route selected by the user from at least two planned routes, and the user's vehicle is controlled to drive along the route selected by the user.

[0182] When a recommended route includes multiple planned routes, it will be understood that the recommendation coefficients for the planned routes included in the recommended route are similar or the same. However, some planned routes may be the shortest in terms of travel time, some may be the most comfortable, and some may be the shortest in terms of distance. In this case, the user may choose a route according to their preferences, thereby providing the user with a good passenger experience.

[0183] In some possible embodiments, perception detection networks, attribute recognition networks, and route evaluation networks may also be deployed on the vehicle. For corresponding descriptions, please refer to the descriptions of the corresponding embodiments. Further details will not be described again.

[0184] In this embodiment of the present application, a perception detection network is deployed on the vehicle, thereby enhancing the vehicle's ability to perceive its surroundings. As a result, the vehicle can perceive surrounding obstacles during the driving process, avoid collisions, and improve vehicle safety. In addition, an attribute recognition network is deployed on the vehicle, thereby enabling the vehicle to further recognize the category of obstacles when they are perceived, thereby improving the vehicle's intelligence.

[0185] Figure 7A is a flowchart of a method for training a perception detection network according to one embodiment of the present application. The method may be applied to the network-side device or components within the network-side device (e.g., a chip or integrated circuit) shown in Figure 1. The perception detection network shown in Figure 2 is used as an example. The perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. The output network includes a plurality of head networks. The method includes, but is not limited to, the following steps.

[0186] S701: In each training process, feature extraction is performed on the image data at each time point within a batch of sensor data by using an image feature extraction network to obtain 3D image features of the image data at K time points.

[0187] For example, one batch of sensor data used in each training process is placed within a sensor dataset.

[0188] For example, one batch of sensor data contains image data at K time points, where the image data at K time points is from at least one camera on the vehicle, and K is a positive integer.

[0189] Specifically, image data from K time points within a batch of sensor data is input to an image feature extraction network. The image feature extraction network then obtains 3D image features of the image data at each time point based on the image data at that time. Thus, the image feature extraction network obtains 3D image features of the image data from K time points and outputs these 3D image features to a feature fusion network.

[0190] For the framework of the image feature extraction network, please refer to the corresponding explanation in the embodiment shown in Figure 2. Further details will not be explained again.

[0191] S702: In each training process, feature extraction is performed on the point cloud data at each time point within a batch of sensor data by using a point cloud feature extraction network to obtain point cloud features of voxels corresponding to the point cloud data at K time points.

[0192] Here, the batch of sensor data further includes point cloud data at K time points, where the point cloud data at K time points are from at least one radar of the vehicle.

[0193] Specifically, point cloud data from K time points within a batch of sensor data is input to a point cloud feature extraction network. Based on the point cloud data at each time point, the point cloud feature extraction network obtains point cloud features for the voxels corresponding to the point cloud data at that time point. Thus, the point cloud feature extraction network obtains point cloud features for the voxels corresponding to the point cloud data at K time points and outputs these point cloud features to a feature fusion network.

[0194] S703: To obtain the fused voxel features of the scene at K time points, a feature fusion network is used to perform feature fusion on the 3D image features of the image data at K time points and the point cloud features of the voxels corresponding to the point cloud data at K time points.

[0195] Here, a scene voxel is a voxel obtained after feature fusion has been performed using a feature fusion network.

[0196] For example, a feature fusion network performs spatial fusion based on the 3D image features of the image data at each time step and the point cloud features of the voxels corresponding to the point cloud data at that time step, thereby obtaining the spatial fusion features of the scene voxels at that time step. Next, the feature fusion network performs temporal fusion based on the spatial fusion features of the scene voxels at K time steps, thereby obtaining the fusion features of the scene voxels at K time steps. In this case, the voxel fusion features of the scene voxels at each time step may also be called the spatiotemporal fusion features of the scene voxels at that time step.

[0197] S704: By using each head network within the output network, K prediction results are output based on the fused voxel features of the scene at K time points, and each of the K prediction results corresponds to the scene at one time point.

[0198] The fused voxel features of the scene at K time points correspond to each head network in the output network, and each head network in the output network performs one prediction task.

[0199] For example, the output network shown in Figure 2 includes head network 1, head network 2, head network 3, and head network 4. Head network 1 is configured to predict corner point information of polygon boxes corresponding to obstacles in the scene, head network 2 is configured to predict the occupancy status of voxels in the scene, head network 3 is configured to predict velocity information of voxels in the scene, and head network 4 is configured to predict the visibility status of voxels in the scene.

[0200] Specifically, in the output network, each head network makes predictions based on the fused voxel features of the scene at each time step in order to obtain prediction results corresponding to the scene at each time step, and as a result, each head network may obtain K prediction results.

[0201] Head network 1 in the output network shown in Figure 2 is used as an example. The K time points are time t1, time t2, ..., and time t K It is assumed that the following are included. Head network 1 outputs prediction result 1 based on the voxel fusion features of the scene at time t1, and prediction result 1 includes corner point information of polygon boxes corresponding to obstacles in the scene at time t1. Head network 1 outputs prediction result 2 based on the voxel fusion features of the scene at time t2, and prediction result 2 includes corner point information of polygon boxes corresponding to obstacles in the scene at time t2, ..., and by analogy, head network 1 outputs K prediction results based on the voxel fusion features of the scene at K times.

[0202] S705: Based on the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network, the loss value of each head network in the output network is obtained.

[0203] Label information corresponding to batches of sensor data is generated by 4D reconstruction based on the batches of sensor data. This label information corresponding to batches of sensor data is used to provide the perception detection network with actual value information corresponding to the perception detection network's prediction results for the batches of sensor data.

[0204] The output network shown in Figure 2 is used as an example. The label information corresponding to a batch of sensor data includes actual value information of the corner points of polygon boxes corresponding to obstacles in the scene at K time points, actual value information of the occupancy status of voxels in the scene at K time points, actual value information of the velocity of voxels in the scene at K time points, and actual value information of the visibility status of voxels in the scene at K time points. The actual value information of the corner points of polygon boxes corresponding to obstacles in the scene at K time points corresponds to K prediction results output by head network 1, the actual value information of the occupancy status of voxels in the scene at K time points corresponds to K prediction results output by head network 2, the actual value information of the velocity of voxels in the scene at K time points corresponds to K prediction results output by head network 3, and the actual value information of the visibility status of voxels in the scene at K time points corresponds to K prediction results output by head network 4.

[0205] The calculation of the loss value of head network 1 is used as an example. The loss value of head network 1 is obtained based on the actual value information of the corner points of polygon boxes corresponding to obstacles in the scene at K time points in the label information and the K prediction results output by head network 1. For example, the loss value of head network 1 at each time point may first be obtained based on the actual value information of the corner points of polygon boxes corresponding to obstacles in the scene at that time and the prediction result at that time from the K prediction results output by head network 1 (i.e., including the corner point information of polygon boxes corresponding to obstacles in the scene at that time), and then the loss value of head network 1 is obtained based on the loss value of head network 1 at each of the K time points. Similarly, another head network in the output network may use this method to calculate the loss value of another head network, and as a result, the loss value of each head network in the output network may be obtained.

[0206] S706: To obtain the loss value corresponding to each training process, the loss value of each head network in the output network is weighted, and the parameters in the perception detection network are updated by using the loss value.

[0207] Here, the weighting of each head network within the output network may be customized by the user.

[0208] After the loss values ​​corresponding to each training process are obtained, the parameters within the perception detection network (e.g., each head network, feature fusion network, image feature extraction network, and point cloud feature extraction network in the output network) are updated using the loss values.

[0209] Figure 7A shows an example where the perceptual detection network is trained separately, and it will be understood that the training procedure for the perceptual detection network is not limited to the form shown in Figure 7A. In some possible embodiments, each head network in the output network within the perceptual detection network may also be trained separately. In this case, S706 does not need to be performed. In some possible embodiments, the perceptual detection network may undergo further joint training with a neural radiance field (NeRF) network to further improve detection accuracy and training efficiency. This is not particularly limited here.

[0210] In this embodiment of the present application, label information corresponding to batches of sensor data in each training process does not need to be generated by manual annotation, thereby reducing labor consumption and improving the efficiency of label information acquisition. The perceptual detection network uses a self-supervised training method, and as a result, it can be seen that perceptual information in a scene in which a vehicle is placed at any given time can be accurately predicted based on the vehicle's input data at that time.

[0211] Figure 7B is a diagram illustrating the training process of a perception detection network according to one embodiment of this application.

[0212] As shown in Figure 7B, one batch of sensor data from the sensor dataset is input to the image feature extraction network and the point cloud feature extraction network within the perception detection network. The image feature extraction network extracts features from the image data within the batch of sensor data to obtain 3D image features of the image data. The point cloud feature extraction network extracts features from the point cloud data within the batch of sensor data to obtain point cloud features of voxels corresponding to the point cloud data. The feature fusion network within the perception detection network fuses the 3D image features of the image data from the image feature extraction network with the point cloud features of voxels corresponding to the point cloud data from the point cloud feature extraction network to obtain fused voxel features of the scene, and inputs these fused voxel features of the scene into the output network within the perception detection network. Each head network within the output network obtains the corresponding prediction result based on the fused voxel features of the scene. The loss value of each head network in the output network is obtained based on the label information corresponding to the prediction results and batches of sensor data output by each head network in the output network. The loss values ​​of each head network in the output network are weighted, and backpropagation is performed using the loss values ​​corresponding to the training process to sequentially update the parameters of the output network, feature fusion network, and feature extraction network (including image feature extraction network and point cloud feature extraction network). Figure 7B is just one example of the training process for a perception detection network, and it should be understood that the training process for a perception detection network is not limited to what is shown in Figure 7B. For example, each head network in the output network may be trained separately.

[0213] For specific details on training the perception detection network, please refer to the relevant description in the embodiment shown in Figure 7A. Further details will not be provided here.

[0214] In some possible embodiments, the attribute recognition network may be trained after the perception detection network has been trained. The training process for the attribute recognition network is described below.

[0215] An attribute recognition network shown in Figure 2 is used as an example. The attribute recognition network includes a text encoding network and an attribute decoding network. For example, the text encoding network may use a trained word vector feature extractor directly (or may be obtained by pre-training text-images). In this embodiment of the present application, only the attribute decoding network may be trained. The trained word vector feature extractor may extract word vector features of text query information based on input text query information, and the word vector features of text query information may represent image semantic features of categories indicated by the text query information.

[0216] For example, the process of acquiring a text coding network by pre-training text images may involve acquiring a large amount of text image training data, where the text image training data includes multiple text image data groups, each text image data group includes text information indicating category information and an image corresponding to the text information, for example, text image training group 1 includes text information indicating a vehicle and an image of a vehicle; inputting the text information in the text image training data into a text encoder to extract the word vector features of each piece of text information; inputting the images in the text information within the text image training data into an image encoder to extract the image features of each image; and adjusting the parameters of the text encoder and the image encoder based on the training idea that the word vector features of the text information and the image features of images belonging to the same text image data group as the text information are as close as possible, and the word vector features of the text information and the image features of images belonging to a different text image data group as possible. The text encoder thus trained may be used directly as a text coding network in an attribute recognition network.

[0217] For example, training an attribute decoding network may involve a text coding network inputting word vector features from multiple received text query pieces into the attribute decoding network, the attribute decoding network predicting category information for each obstacle based on the word vector features from the multiple text query pieces and the fused voxel features of the obstacles provided by the perception detection network, obtaining the training loss value for the attribute decoding network based on the predicted category information for each obstacle and the category annotation information for each obstacle, and then inversely updating the parameters of the attribute decoding network based on the training loss value for the attribute decoding network.

[0218] In some possible embodiments, the path evaluation network may be trained after the perception detection network has been trained. The training process for the path evaluation network is described below.

[0219] The route evaluation network shown in Figure 2 is used as an example. The route evaluation network includes a route coding network, a feature interaction network, and an evaluation output network. For example, the training process for a route evaluation network may involve acquiring route training data, which includes multiple routes planned by a vehicle within K time ranges and recommendation coefficient annotation information for the multiple routes; inputting the multiple routes into a route coding network, which outputs the extracted route features for each route to a feature interaction network, which outputs the risk features for each route based on the route features and scene voxel fusion features of each route at K time points (from the perception detection network); the evaluation output network outputs the predicted recommendation coefficients for the multiple routes based on the risk features of the multiple routes; and determining the predicted recommended route from the multiple routes based on the predicted recommendation coefficients for the multiple routes; and acquiring the loss values ​​for the multiple routes based on the predicted recommendation coefficients and recommendation coefficient annotation information for the multiple routes, which are obtained by weighting the respective loss values ​​of the multiple routes; and updating the parameters in the route evaluation network based on the loss values ​​for the multiple routes.

[0220] It can be seen that the training of the perception detection network, attribute recognition network, and path evaluation network is performed separately and independently. In some possible embodiments, the perception detection network, attribute recognition network, and path evaluation network may be trained jointly. In this case, the loss value corresponding to each training process is obtained by weighting the loss value of the attribute recognition network in the training process, the loss value of the perception detection network in the training process (i.e., the loss value of the attribute decoding network in the training process), and the loss value of the path evaluation network in the training process. Finally, the parameters in the perception detection network, the attribute decoding network in the attribute recognition network, and the path evaluation network may be updated separately based on the loss values ​​in the training process.

[0221] Figure 8 shows a hardware structure of a chip according to one embodiment of the present application. The chip may be configured to perform an intelligent driving method and / or training method according to an embodiment of the present application.

[0222] As shown in Figure 8, a neural network processing unit (NPU) 80 is attached to a host CPU acting as a coprocessor, and the Host CPU assigns tasks to perform the processes related to the intelligent driving method or the training method in the above-described embodiment.

[0223] The core of the NPU is the arithmetic circuit 803, and the controller 804 controls the arithmetic circuit 803 to extract data from memory (weighted memory or input memory) and perform calculations.

[0224] In some embodiments, the arithmetic circuit 803 includes multiple process elements (process engines, PEs). In some embodiments, the arithmetic circuit 803 is a two-dimensional systolic array. Alternatively, the arithmetic circuit 803 may be a one-dimensional systolic array or another electronic circuit capable of performing mathematical operations such as multiplication and addition. In some embodiments, the arithmetic circuit 803 is a general-purpose matrix processor.

[0225] For example, suppose we have an input matrix A, a weighting matrix B, and an output matrix C. The arithmetic circuit extracts the corresponding data for matrix B from the weighting memory 802 and buffers the corresponding data in each PE within the arithmetic circuit. The arithmetic circuit extracts the data for matrix A from the input memory 801, performs matrix operations between the data of matrix A and matrix B to obtain a submatrix result or a final matrix result, and stores the result in the accumulator 808.

[0226] The vector computation unit 807 may perform further processing on the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operations, logarithmic operations, and value comparisons. For example, the vector computation unit 807 may be used for network computations of non-convolutional / non-FC layers within a neural network, such as pooling, batch normalization, and local response normalization.

[0227] In some embodiments, the vector computation unit 807 can store the processed output vector in the integrated memory 806. For example, the vector computation unit 807 may apply a nonlinear function, such as a vector of cumulative values, to the output of the arithmetic circuit 803 to generate an activation value. In some embodiments, the vector computation unit 807 generates a normalized value, a combined value, or a normalized value and a combined value. In some embodiments, the processed output vector can be used, for example, as an activation input to the arithmetic circuit 803, which is used by subsequent layers in the neural network.

[0228] The integrated memory 806 is configured to store input and output data.

[0229] The direct memory access controller (DMAC) transfers input data from external memory to input memory 801 and / or integrated memory 806, stores weighted data from external memory in weighted memory 802, and stores data from integrated memory 806 in external memory.

[0230] The bus interface unit (BIU) 810 is configured to enable interaction between the host CPU, DMAC, and instruction fetch buffer 809 via the bus.

[0231] An instruction fetch buffer 809 connected to the controller 804 is configured to store instructions that should be used by the controller 804.

[0232] The controller 804 is configured to control the work process of the arithmetic accelerator by calling instructions buffered in the instruction fetch buffer 809.

[0233] Typically, the integrated memory 806, input memory 801, weighted memory 802, and instruction fetch buffer 809 are all on-chip memory, while external memory is memory outside the NPU. External memory may be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other read / write memory.

[0234] Figure 9A is a diagram showing the structure of a computing device according to one embodiment of the present application. The computing device 30 includes a receiving unit 310 and a processing unit 312. The computing device 30 may be implemented using hardware, software, or a combination of software and hardware.

[0235] The receiving unit 310 is configured to acquire sensor data for a first scene, the sensors including at least one of a camera and a radar. The processing unit 312 is configured to input the acquired data to a perception detection network and output perception information, the perception information indicating voxels of obstacles in the first scene. The processing unit 312 is further configured to display voxels of obstacles based at least on the perception information.

[0236] In some possible embodiments, the computing device 30 further includes a display unit 314 (not shown). The display unit 314 is configured to display obstacles based on perceptual information, where the obstacles are marked using polygon boxes and / or display voxels of the obstacles based on perceptual information.

[0237] The computing device 30 may be configured to perform the method described in the embodiment of Figure 4. In the embodiment of Figure 4, the receiving unit 310 may be configured to perform S401, and the processing unit 312 may be configured to perform S402 and S403.

[0238] Figure 9B is a diagram showing the structure of a training device according to one embodiment of the present application. The training device 40 includes an encoding unit 410, a decoding unit 412, and an update unit 414. The training device 40 may be implemented using hardware, software, or a combination of software and hardware.

[0239] The encoding unit 410 is configured to perform feature extraction on image data at each time point within a batch of sensor data in each training process by using an image feature extraction network to obtain 3D image features of image data at K time points, where K is a positive integer. In each training process, it performs feature extraction on point cloud data at each time point within a batch of sensor data by using a point cloud feature extraction network to obtain point cloud features of voxels corresponding to point cloud data at K time points, and performs feature fusion on the 3D image features of image data at K time points and the point cloud features of voxels corresponding to point cloud data at K time points by using a feature fusion network to obtain fused features of scene voxels at K time points. The decoding unit 412 is configured to output K prediction results based on the fused features of scene voxels at K time points by using each head network in the output network, where each of the K prediction results corresponds to a scene at one time point. The update unit 414 is configured to update the parameters in the perception detection network by obtaining the loss value of each head network in the output network based on the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network, weighting the loss values ​​of each head network in the output network to obtain the loss value corresponding to each training process, and using the loss values.

[0240] The training device 40 may be configured to perform the method described in the embodiment of Figure 7A. In the embodiment of Figure 7A, the encoding unit 410 may be configured to perform S701 to S703, the decoding unit 412 may be configured to perform S704, and the update unit 414 may be configured to perform S705 and S706.

[0241] It should be understood that the division of units in the aforementioned devices (e.g., computing device 30 and training device 40) is merely a logical functional division. In actual implementation, all or part of the units may be integrated into a single physical entity or they may be physically separated. In addition, units within a device may be realized in the form of a processor calling software. For example, the device includes a processor, the processor is connected to memory, the memory stores instructions, and the processor calls the instructions stored in memory to perform one of the aforementioned methods or to realize the functionality of a unit within the device. The processor may be, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory may be memory within the device or memory outside the device. Alternatively, units within a device may be realized in the form of hardware circuits, and some or all of the functionality of a unit may be realized by designing a hardware circuit. A hardware circuit may be understood as one or more processors. For example, in one embodiment, the hardware circuit is an application-specific integrated circuit (ASIC), and some or all of the functionality of a unit is realized by designing the logical relationships between elements in the circuit. As another example, in another embodiment, the hardware circuitry may be implemented using programmable logic devices (PLDs), for example, a field programmable gate array (FPGA) which may contain a large number of logic gate circuits, and the connections between the logic gate circuits are configured using a configuration file to implement some or all of the functions of the unit. All units of the device may be implemented in the form of software invoked by a processor or in the form of hardware circuitry, or some units may be implemented in the form of software invoked by a processor and the remaining units may be implemented in the form of hardware circuitry.

[0242] In this embodiment of the present application, the processor is a circuit having signal processing capabilities. In one embodiment, the processor may be a circuit having instruction readout and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP). In another embodiment, the processor may implement specific functions by using logic relationships of hardware circuits. The logic relationships of hardware circuits may be fixed or reconfigurable. For example, the processor is an application-specific integrated circuit (ASIC) or an FPGA implemented by hardware circuits, such as a programmable logic device (PLD). In reconfigurable hardware circuits, the process by which the processor loads configuration documents to implement a hardware circuit configuration may be understood as the process by which the processor loads instructions to implement some or all of the functions of the unit. In addition, the circuit may be a hardware circuit designed for artificial intelligence, or it may be understood as an ASIC, for example, a neural network processing unit (NPU), a tensor processing unit (TPU), or a deep learning processing unit (DPU).

[0243] It can be seen that a unit within the device may be configured as one or more processors (or processing circuits) for carrying out the method described above, for example, as a CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0244] In addition, all or some of the units within the device may be integrated, or each unit within the device may be implemented independently. In one implementation configuration, the units may be integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor configured to perform one of the methods described above or to realize the functions of the units of the device. The type of at least one processor may vary, including, for example, a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, and so on.

[0245] Figure 10 is a diagram illustrating the structure of a processing device according to one embodiment of the present application. As shown in Figure 10, the processing device 50 includes a processor 501, a communication interface 502, a memory 503, and a bus 504. The processor 501, the memory 503, and the communication interface 502 communicate with each other via the bus 504. It should be understood that the number of processors and memories in the processing device 50 is not limited in this application.

[0246] In one embodiment, the processing device 50 is a component (e.g., a chip or integrated circuit) used for the automated driving control of the vehicle. The vehicle is configured using an automated driving system, where the automated driving system is not limited to a fully automated driving system, a highly automated driving system, a conditionally automated driving system, a partially automated driving system, etc. Those skilled in the art will understand that all non-fully manual driving systems for intelligent driving may be included in this concept.

[0247] In another embodiment, the processing device 50 may be a network-side device. The network-side device is a device having computing capabilities. The network-side device may be, for example, a server deployed on the network side (e.g., a server for intelligent operation processing), or a component or chip within a server. In some possible embodiments, the network-side device may also be a system-level device including multiple servers. The network-side device may be deployed in a cloud environment or an edge environment. This is not particularly limited here.

[0248] Bus 504 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. Buses can be classified into address buses, data buses, control buses, etc. For ease of representation, in Figure 8, buses are represented by using only one line. However, this does not mean that there is only one bus or only one type of bus. Bus 504 may include paths for transferring information between components of the processing device 50 (e.g., memory 503, processor 501, and communication interface 502).

[0249] For details regarding processor 501, please refer to the relevant description of the processor in the above embodiment. Further details will not be explained again.

[0250] Memory 503 is configured to provide a storage space that can store data such as an operating system and computer programs. Memory 503 may be one or a combination of random access memory (RAM), erasable programmable read-only memory (EPROM), read-only memory (ROM), compact disc read memory (CD-ROM), etc. Memory 503 may exist independently or may be integrated into the processor 501.

[0251] The communication interface 502 may be configured to provide information input or output for the processor 501. Alternatively, the communication interface 502 may be configured to receive data transmitted from an external source and / or transmit data to an external source, and may be a wired link interface such as an Ethernet cable, or a wireless link interface (such as Wi-Fi, Bluetooth, Universal Wireless Transmission, etc.). Alternatively, the communication interface 502 may further include a transmitter (e.g., a radio frequency transmitter or antenna), a receiver, etc., coupled to the interface.

[0252] In some possible embodiments, the processing device 50 further includes a display 505. The display 505 and the processor 501 are connected or coupled via a bus 504. The display 505 may be configured to display polygon instances of a first scene. The display 505 may be a display screen, and the display screen may be a liquid crystal display (LCD), an organic or inorganic light-emitting diode (OLED), an active matrix / organic light-emitting diode (AMOLED), etc. The display 505 may also be an in-car infotainment tablet, an in-car display, a head-up display (HUD) system, an augmented reality head-up display (AR-HUD) system, etc.

[0253] The processor 501 within the processing device 50 is configured to read a computer program stored in the memory 503 and execute the aforementioned communication method, for example, the method described in Figure 4 or Figure 7A.

[0254] In possible design configurations, the processing device 50 may be one or more modules within an execution entity for performing the method shown in Figure 4, and the processor 501 may read one or more computer programs stored in memory and perform the following operations, namely: An operation to acquire sensor data for a first scene by using the receiving unit 310, wherein the sensor includes at least one of a camera and a radar, and An operation in which collected data is input into a perception detection network and perceptual information is output, wherein the perceptual information indicates voxels of obstacles in a first scene, and an operation in which voxels of obstacles are displayed based on the perceptual information. It may be configured to do so.

[0255] In another possible design, the processing device 50 may be one or more modules within an execution entity for performing the method shown in Figure 7A, and the processor 501 may read one or more computer programs stored in memory and perform the following operations, namely: The encoding unit 410 performs feature extraction on image data at each time point within a batch of sensor data by using an image feature extraction network to obtain 3D image features of image data at K time points in each training process, where K is a positive integer. It also performs feature extraction on point cloud data at each time point within a batch of sensor data by using a point cloud feature extraction network to obtain point cloud features of voxels corresponding to point cloud data at K time points, and performs feature fusion on the 3D image features of image data at K time points and the point cloud features of voxels corresponding to point cloud data at K time points by using a feature fusion network to obtain fused features of scene voxels at K time points. The decoding unit 412 outputs K prediction results based on the fused voxel features of the scene at K time points by using each head network in the output network, wherein each of the K prediction results corresponds to the scene at one time point. The update unit 414 obtains the loss value of each head network in the output network based on the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network, weights the loss value of each head network in the output network to obtain the loss value corresponding to each training process, and updates the parameters in the perception detection network by using the loss value. It may be configured to do so.

[0256] In the embodiments described herein, each embodiment has its own focus. For aspects not described in detail in one embodiment, please refer to the relevant description in another embodiment. Furthermore, in the embodiments of this application, unless otherwise specified or logically contradictory, the terminology and / or descriptions of the various embodiments are consistent and can be referenced to one another. Based on the internal logical relationships between the technical features, the technical features of the various embodiments may be combined to form new embodiments.

[0257] It should be noted that those skilled in the art will see that all or part of the steps in each of the embodiments described above can be implemented by a program that instructs the relevant hardware. The program may be stored on a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disc memory, magnetic disk memory, magnetic tape memory, or any other computer-readable medium that can be configured to carry or store data.

[0258] Essentially, the technical solutions of this application, or a contributing portion thereof, or all or part of the technical solutions, may be implemented in the form of a software product. The computer program product is stored on a storage medium and includes several instructions for instructing a device (which may be a personal computer, server, network device, robot, single-chip microcomputer, chip, robot, etc.) to perform all or part of the steps of the method described in the embodiments of this application. [Explanation of symbols]

[0259] 30 Computing devices 40 Training equipment 50 Processing Devices 310 Receiving Unit 312 Processing Units 314 Display Unit 410 encoding units 412 Decoding Units 414 Renewal Unit 501 Processor 502 Communication Interface 503 memory 504 Bus 505 displays 801 Input Memory 802 Weighted Memory 803 Arithmetic circuit 804 Controller 805 Direct Memory Access Controller 806 Integrated Memory 807 Vector Computation Unit 809 Instruction fetch buffer

Claims

1. An intelligent driving method, A step of acquiring sensor data for a first scene, wherein the sensor includes at least one of a camera and a radar, The steps include inputting the collected data into a perception detection network and outputting perception information, wherein the perception information indicates voxels of obstacles in the first scene, The steps include controlling the operation of the vehicle based on at least the aforementioned perceptual information, and Methods that include...

2. The aforementioned method, A step of displaying the obstacle based on the aforementioned perceptual information, wherein the obstacle is marked by a polygon box, and / or The step of displaying the voxels of the obstacle based on the aforementioned perceptual information. The method according to claim 1, further comprising:

3. The aforementioned perceptual information includes the following information, namely: The occupancy status of the voxel in the first scene, the speed information of the voxel in the first scene, the visibility status of the voxel in the first scene, and the corner point information of the polygon box corresponding to the obstacle. It includes at least one of the following: The method according to claim 1 or 2, wherein the polygon box corresponding to the obstacle is associated with the voxel of the obstacle.

4. The perceptual information further indicates voxels on the road surface in the first scene, and the step of controlling the driving of the vehicle based on at least the perceptual information is, The steps include generating road surface shape information of the first scene based on at least the perceptual information, A step of adjusting the suspension inside the vehicle based on the road surface shape information. The method according to any one of claims 1 to 3, including the method described in any one of claims 1 to 3.

5. The method according to any one of claims 1 to 4, wherein the step of controlling the driving of a vehicle based on at least the perceptual information includes the step of adjusting the driving path of the vehicle based on at least the perceptual information, the adjusted driving path does not pass through the area in which the voxels of the obstacle are located.

6. The collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. The image feature extraction network is configured to extract 3D image features from the image data. The point cloud feature extraction network is configured to extract point cloud features of voxels corresponding to the point cloud data. The feature fusion network is configured to perform fusion based on the 3D image features and the point cloud features of the voxels corresponding to the point cloud data to obtain the fused features of the voxels in the first scene. The method according to any one of claims 1 to 5, wherein the output network is configured to process the fused features of the voxels in the first scene and to output the perceptual information.

7. The aforementioned method, A step comprising inputting text query information and the fused voxel features of the obstacle into an attribute recognition network and outputting category information of the obstacle, wherein the text query information is used to request a category query, and A step of displaying the category information of the aforementioned obstacles. It further includes, The method according to claim 6, wherein the fusion features of the voxels of the obstacle are determined based on corner point information of a polygon box corresponding to the obstacle and the fusion features of the voxels of the first scene, and the polygon box corresponding to the obstacle is associated with the voxels of the obstacle.

8. The aforementioned method, The steps include obtaining multiple planned routes for the vehicle, A step of inputting the plurality of planned routes of the vehicle and the fused features of the voxels of the first scene into a route evaluation network, and outputting recommendation coefficients for the plurality of planned routes and recommended routes in the plurality of planned routes, wherein the recommended routes are associated with the recommendation coefficients for the plurality of planned routes, The steps include displaying the recommended route and The method according to claim 6 or 7, further comprising:

9. A system for intelligent driving, A perception detection network configured to output perceptual information based on sensor data for a first scene, wherein the perceptual information indicates voxels of obstacles in the first scene, and the sensor includes at least one of a camera and a radar. An attribute recognition network configured to output category information of an obstacle based on text query information and the fusion features of the voxels of the obstacle, wherein the fusion features of the voxels of the obstacle are determined based on corner point information of a polygon box corresponding to the obstacle and the fusion features of the voxels of the first scene, the polygon box corresponding to the obstacle is associated with the voxels of the obstacle, and the fusion features of the voxels of the first scene are acquired by the perception detection network by performing temporal and / or spatial fusion based on at least one of the 3D image features and point cloud features of the voxels extracted from the collected data, A route evaluation network configured to output recommendation coefficients for a plurality of planned routes and recommended routes in a plurality of planned routes based on a plurality of planned routes and the fusion features of the voxels in the first scene, wherein the recommended routes are associated with the recommendation coefficients of the plurality of planned routes, and the route evaluation network and A system equipped with these features.

10. The aforementioned perceptual information includes the following information, namely: The occupancy status of the voxel in the first scene, the speed information of the voxel in the first scene, the visibility status of the voxel in the first scene, and the corner point information of the polygon box corresponding to the obstacle. It includes at least one of the following: The system according to claim 9, wherein the polygon box corresponding to the obstacle is associated with the voxel of the obstacle.

11. The collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. The image feature extraction network is configured to extract the 3D image features of the image data, The point cloud feature extraction network is configured to extract the point cloud features of the voxels corresponding to the point cloud data. The feature fusion network is configured to perform fusion based on the 3D image features and the point cloud features of the voxels corresponding to the point cloud data to obtain the fused features of the voxels in the first scene. The system according to claim 9 or 10, wherein the output network is configured to process the fused features of the voxels in the first scene and to output the perceptual information.

12. The attribute recognition network comprises a text encoding network and an attribute decoding network. The text coding network is configured to extract word vector features from the text query information, The system according to any one of claims 9 to 11, wherein the attribute decoding network is configured to output the category information of the obstacle based on the word vector features and the fusion features of the voxels of the obstacle.

13. The aforementioned path evaluation network comprises a path coding network, a feature interaction network, and an evaluation output network. The route coding network is configured to extract the route features of each of the multiple planned routes. The feature interaction network is configured to acquire risk features for each planned path based on the path features of each planned path and the fusion features of the voxels in the first scene. The system according to any one of claims 9 to 12, wherein the evaluation output network is configured to output the recommendation coefficients for the plurality of planned routes and the recommended routes for the plurality of planned routes based on the risk characteristics of the plurality of planned routes.

14. A device for intelligent driving, A receiving unit configured to acquire sensor data for a first scene, wherein the sensor includes at least one of a camera and a radar, A processing unit configured to input the collected data into a perception detection network and output perception information, wherein the perception information indicates voxels of obstacles in the first scene, and Equipped with, The processing unit is further configured to control the operation of a vehicle based at least on the perceptual information.

15. A computing device comprising memory and a processor, wherein the memory is configured to store program instructions, and when the processor executes the program instructions in the memory, the computing device performs the method according to any one of claims 1 to 8.

16. A vehicle comprising a system according to any one of claims 9 to 13, or a device according to claim 14 or 15.

17. A computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and the program instructions are used to perform the method according to any one of claims 1 to 8.

Images