Intelligent driving method and apparatus

Abstract

Disclosed are an intelligent driving method and apparatus, the method comprising: acquiring data collected by a sensor of a vehicle for a scene, the sensor comprising at least one of a camera and a radar; inputting the collected data into a perception detection network, and outputting perception information, the perception information being used for indicating a voxel of an obstacle in a first scene; and at least on the basis of the perception information, controlling the vehicle to drive. In this way, the perception capability of the vehicle for the surrounding environment can be enhanced, thereby improving the accuracy in detecting an obstacle, and preventing collisions.

Description

Intelligent driving method and device Technical Field

[0001] The present application relates to the field of intelligent driving, and in particular to an intelligent driving method and device. Background Art

[0002] The ability of an autonomous vehicle's perception system to perceive the surrounding environment is closely related to the vehicle's safe driving.

[0003] Currently, the perception systems of some vehicles use a purely visual approach to detect obstacles in the surrounding environment. This approach is highly dependent on training materials (such as whitelists). The perception system must be trained on obstacles before it can recognize them.

[0004] Currently, some vehicle perception systems use lidar or millimeter-wave radar to detect obstacles in the surrounding environment, but this method of detection is easily affected by weather. For example, the accuracy of obstacle detection is low in rainy and snowy weather.

[0005] Summary of the Invention

[0006] The present application discloses an intelligent driving method and device, which can enhance a vehicle's perception of surrounding objects, thereby improving the accuracy of obstacle detection and avoiding collisions.

[0007] In a first aspect, the present application provides an intelligent driving method, which includes: obtaining data collected by a sensor on a first scene, the sensor including at least one of a camera and a radar; inputting the collected data into a perception detection network, and outputting perception information, wherein the perception information is used to indicate voxels of obstacles in the first scene; and controlling vehicle driving based at least on the perception information.

[0008] Here, obstacles refer to entities that a vehicle is not expected to collide with during driving. Such entities can be static or dynamic. Static entities can include, for example, cardboard boxes on the road, road construction signs, road dividing railings, dirt piles, tires, overturned vehicles, lying people, animals, buildings beside the road, trees, parked vehicles, road signs, telephone poles, roadside medians, and other stationary objects with volume and mass. Dynamic entities can include, for example, pedestrians (e.g., walking pedestrians, cyclists, etc.), animals, vehicles, and vehicles carrying cargo (e.g., loaded with cardboard boxes, branches, or other cargo).

[0009] It is understood that this application does not limit the physical form of obstacles. For example, a vehicle can be in a state of being driven or parked with its tires touching the ground, or in a state of being overturned after a collision, or with cargo (such as branches or cartons) loaded into its trunk, or even when multiple vehicles are connected. The application also does not limit the type of vehicle when the obstacle is a vehicle. Vehicle types can include cars, trucks, buses, trailers, incomplete vehicles, motorcycles, bicycles, and so on.

[0010] Here, the radar includes at least one of a laser radar and a millimeter-wave radar.

[0011] For example, the first scene can be understood as the environment space that can be detected by the sensors on the vehicle during the vehicle's driving process. It can be understood that during the vehicle's driving process, each moment can correspond to a scene, and the scenes corresponding to multiple moments include the scenes corresponding to each moment in the multiple moments.

[0012] Exemplarily, the perception detection network is trained based on a sensor dataset and label information corresponding to the sensor dataset generated using 4D reconstruction (i.e., spatiotemporal reconstruction of both moving and static objects). This label information is used to provide the perception detection network with ground truth information for its predictions during training. It is understood that 4D reconstruction can describe the changes of physical objects in three-dimensional space in the temporal dimension.

[0013] For example, in addition to controlling vehicle driving based on perception information, vehicle driving can also be controlled by combining at least one of navigation map information, high-precision map information, roadside equipment, and live traffic information broadcast by surrounding vehicles.

[0014] For example, the method can be applied to a vehicle or a component within the vehicle for intelligent driving control (e.g., a chip or integrated circuit). The vehicle is equipped with an autonomous driving system. Here, the autonomous driving system is not limited to a fully autonomous driving system, a highly autonomous driving system, a conditional autonomous driving system, or a partially autonomous driving system. Those skilled in the art will understand that any non-fully manual driving system that provides intelligent driving can be covered by this concept.

[0015] This method uses pure vision or a combination of vision and radar to collect scene data. This data is then processed by a perception detection network to output voxel-based perception information indicating obstacles. This enhances the vehicle's ability to perceive surrounding objects, enabling the perception of obstacles independent of semantic categories, and improving the generalization and accuracy of obstacle detection in a scene. Furthermore, controlling vehicle driving based on this perception information can enhance driving safety.

[0016] In a possible implementation manner of the first aspect, the method further includes: displaying the obstacle based on the perception information, the obstacle being marked with a polygonal box; and / or displaying the voxels of the obstacle based on the perception information, the voxels of the obstacle being marked with a polygonal box.

[0017] Illustratively, the polygonal frame may be two-dimensional or three-dimensional.

[0018] For example, when displaying obstacles or voxels of obstacles, dynamic obstacles and static obstacles at the current moment can be distinguished by different colors, or by displaying an arrow on the dynamic obstacle to distinguish dynamic obstacles from static obstacles. The arrow on the dynamic obstacle is used to indicate the direction of movement of the dynamic obstacle.

[0019] By implementing the above implementation method, marking obstacles with polygonal boxes better fits the shape of the obstacles themselves. By presenting the obstacles and / or their voxels, users can clearly and intuitively understand the vehicle's current perception of the surrounding environment.

[0020] In a possible implementation of the first aspect, the perception information includes at least one item of the following information: the occupancy status of the voxels of the first scene, the speed information of the voxels of the first scene, the visibility status of the voxels of the first scene, and the corner point information of the polygonal box corresponding to the obstacle; wherein the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

[0021] Here, the visibility state of a voxel can be categorized as "visible" or "invisible." For example, in a scene currently located by a vehicle, if a voxel in the scene is not detected by any sensor (including cameras and radar) on the vehicle at the current moment, the voxel's visibility state is "invisible." If the voxel is detected by at least one sensor on the vehicle, the voxel's visibility state is "visible."

[0022] Here, the occupancy state of a voxel can be categorized as "occupied" or "empty" (i.e., unoccupied). For example, in the scene currently occupied by a vehicle, if a voxel in the scene has a physical entity at the corresponding spatial location in the physical world of the scene, the occupancy state of the voxel is "occupied." If no physical entity exists at the corresponding spatial location in the physical world of the scene, the occupancy state of the voxel is "empty." It is understood that air is not a physical entity.

[0023] For example, associating the polygonal box corresponding to the obstacle with the voxels of the obstacle can be understood as: corner point information of the polygonal box corresponding to the obstacle is obtained based on the index information of the voxels of the obstacle. The corner point information of the polygonal box corresponding to the obstacle can be calculated based on the index information of the voxels of the obstacle using a convex hull algorithm, for example.

[0024] By implementing the above implementation method, the blind spots of the vehicle in the current scene can be known based on the visibility status of the voxels, the areas where the vehicle should avoid the voxels in the "occupied" state to avoid collisions based on the occupancy status of the voxels, the obstacles in the scene can be quickly located based on the corner point information of the polygonal box corresponding to the obstacle, and the speed information of the obstacle in the current scene can be determined based on the speed information of the voxels and the corner point information of the polygonal box corresponding to the obstacle.

[0025] In a possible implementation of the first aspect, the perception information is also used to indicate voxels of the road surface of the first scene, and the controlling of vehicle driving at least based on the perception information includes: generating road surface geometry information of the first scene at least based on the perception information; and adjusting the suspension in the vehicle based on the road surface geometry information.

[0026] Exemplarily, the road surface geometry information is used to indicate the road surface condition of the first scene (eg, whether there are potholes on the road surface, whether there are bumps on the road surface, etc.).

[0027] By implementing the above-mentioned implementation method, the vehicle can obtain the road conditions in front of the vehicle in advance based on the perception information. When it detects that the road ahead is undulating, the vehicle has enough time to adjust the vehicle's suspension in time to keep the vehicle as level and stable as possible during driving, reducing the vibration caused by the undulations in the road and improving the comfort of riding in the vehicle.

[0028] In a possible implementation of the first aspect, controlling the vehicle driving at least based on the perception information includes: adjusting the vehicle's driving path at least according to the perception information, the adjusted driving path not passing through the area where the voxel of the obstacle is located.

[0029] The implementation of the above-mentioned implementation method can avoid collisions between the vehicle and obstacles during driving, which is beneficial to improving the safety of vehicle driving.

[0030] In a possible implementation of the first aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network, wherein:

[0031] The image feature extraction network is used to extract 3D image features of the image data;

[0032] The point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data;

[0033] The feature fusion network is used to fuse the 3D image features with the point cloud features of the voxels corresponding to the point cloud data to obtain fused features of the voxels of the first scene;

[0034] The output network is used to process the fused features of the voxels of the first scene and output the perception information.

[0035] Here, different from the voxels corresponding to the point cloud data, in this application, the voxels of the first scene refer to the voxels fused by the feature fusion network.

[0036] By implementing the above-mentioned implementation method, the vehicle can perceive obstacles in the surrounding environment through the raw data of the multimodal sensors on the vehicle, integrating the advantages of different sensors (such as providing texture semantic information of the image and depth information of the point cloud), which is conducive to enhancing the vehicle's perception of the surrounding environment and improving the generalization ability and accuracy of obstacle detection.

[0037] In a possible implementation of the first aspect, the method further includes: inputting text query information and fused features of the voxels of the obstacle into an attribute recognition network, and outputting category information of the obstacle; the text query information is used to request a query category; and displaying the category information of the obstacle; wherein the fused features of the voxels of the obstacle are determined based on corner point information of a polygonal box corresponding to the obstacle and fused features of the voxels of the first scene, and the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

[0038] For example, the text query information is used to request a query for Q categories. Assume that the number of obstacle categories in a scene is P, where Q and P are positive integers and Q is greater than P. In other words, the number of categories that the attribute recognition network actually supports is greater than the number of obstacle categories in any scene. This ensures that the attribute recognition network avoids omissions when identifying the categories of obstacles in any scene.

[0039] Exemplarily, the fusion features of the voxels of the obstacle are determined based on the corner point information of the polygonal box corresponding to the obstacle and the fusion features of the voxels of the first scene, which means that: since the corner point information of the polygonal box corresponding to the obstacle corresponds to the index information of the voxels of the obstacle, the fusion features of the voxels of the obstacle can be determined from the fusion features of the voxels of the first scene based on the index information of the voxels of the obstacle.

[0040] Here, associating the polygonal box corresponding to the obstacle with the voxels of the obstacle means that the corner point information of the polygonal box corresponding to the obstacle is obtained based on the index information of the voxels of the obstacle.

[0041] Exemplarily, attribute recognition includes a text encoding network and an attribute decoding network, wherein the text encoding network is used to extract word vector features of text query information; the attribute decoding network is used to output the category information of the obstacle based on the fusion features of the word vector features and the voxels of the obstacle.

[0042] By implementing the above implementation method, on the basis of the perception detection network, by deploying the attribute recognition network, the vehicle can not only detect obstacles in the surrounding environment during driving, but also identify the category of obstacles, so that the vehicle can not only see objects but also understand them.

[0043] In a possible implementation of the first aspect, the method further includes: obtaining multiple planned paths for a vehicle; inputting the fusion features of the multiple planned paths of the vehicle and the voxels of the first scene into a path evaluation network, outputting recommendation coefficients of the multiple planned paths and a recommended path among the multiple planned paths, wherein the recommended path is associated with the recommendation coefficients of the multiple planned paths; and displaying the recommended path.

[0044] Exemplarily, the path evaluation network includes a path encoding network, a feature interaction network and an evaluation output network, wherein the path encoding network is used to extract the path features of each planned path among multiple planned paths; the feature interaction network is used to obtain the risk features of each planned path based on the path features of each planned path and the fusion features of the voxels of the first scene; and the evaluation output network is used to output the recommendation coefficients of the multiple planned paths and the recommended paths among the multiple planned paths based on the risk features of the multiple planned paths.

[0045] Exemplarily, the recommended path is the planned path corresponding to the highest recommendation coefficient among multiple planned paths.

[0046] Exemplarily, the recommendation coefficient of the planned path can be obtained based on at least one of the risk coefficient, comfort and traffic efficiency of the planned path. Among them, the risk coefficient of the planned path is related to at least one of the factors such as the distance between the planned path and obstacles (including visible obstacles and obstacles currently in blind spots), whether the planned path conflicts with the paths of other traffic participants on the road (for example, whether a collision will occur at the current moment or in the future), etc. The traffic efficiency of the planned path is related to at least one of the factors such as the length of the planned path, the estimated travel time corresponding to the planned path, the number of traffic lights along the planned path, and the area of ​​the drivable area where the planned path is located. The comfort of the planned path is related to at least one of the factors such as the magnitude and frequency of the steering acceleration of the planned path, the rate of change of the acceleration of the planned path, the flatness of the road surface along the planned path, the number of traffic lights along the planned path, the type of road where the planned path is located, and whether the area along the planned path is shady.

[0047] For example, when other factors remain unchanged, the lower the risk coefficient of the planned path, the higher the recommendation coefficient of the planned path; when other factors remain unchanged, the higher the comfort of the planned path, the higher the recommendation coefficient of the planned path; when other factors remain unchanged, the higher the traffic efficiency of the planned path, the higher the recommendation coefficient of the planned path.

[0048] By implementing the above implementation method, on the basis of the perception detection network, the attribute recognition network is deployed to realize route recommendation, which is beneficial to improving the safety and comfort of vehicle driving.

[0049] In a second aspect, the present application provides a system for intelligent driving, the system comprising: a perception detection network, configured to output perception information based on data collected by a sensor for a first scene, the perception information being used to indicate voxels of obstacles in the first scene; the sensor comprising at least one of a camera and a radar; an attribute recognition network, configured to output category information of the obstacle based on text query information and fusion features of the voxels of the obstacle, the fusion features of the voxels of the obstacle being determined based on corner point information of a polygonal box corresponding to the obstacle and fusion features of the voxels of the first scene, the polygonal box corresponding to the obstacle being associated with the voxels of the obstacle, the fusion features of the voxels of the first scene being obtained by temporally and / or spatially fusing at least one of 3D image features and voxel point cloud features extracted from the collected data by the perception detection network; and a path evaluation network, configured to output recommendation coefficients of the multiple planned paths and recommended paths among the multiple planned paths based on the fusion features of the voxels of the first scene, the recommended paths being associated with the recommendation coefficients of the multiple planned paths.

[0050] For example, the system can be deployed in a vehicle or a component within the vehicle for intelligent driving control, which can be, for example, a chip or an integrated circuit. The vehicle can be specifically described in the first aspect above and will not be repeated here.

[0051] In the above method, the perception detection network can enhance the system's ability to perceive the surrounding environment used for intelligent driving, thereby avoiding collisions between the system's deployment end and obstacles, improving the safety of the system; the attribute recognition network enables the system to identify the category of obstacles based on the perception of obstacles, thereby improving the intelligence of the system; the path evaluation network can achieve the recommendation of low-risk paths, providing convenience for intelligent travel.

[0052] The beneficial effects of any feature of the second aspect described below can be referred to the description of the beneficial effects of the corresponding feature of the first aspect described above, and will not be repeated here.

[0053] In a possible implementation of the second aspect, the perception information includes at least one item of the following information: the occupancy status of the voxels of the first scene, the speed information of the voxels of the first scene, the visibility status of the voxels of the first scene, and the corner point information of the polygonal box corresponding to the obstacle; wherein the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

[0054] In a possible implementation of the second aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network, wherein:

[0055] The image feature extraction network is used to extract 3D image features of the image data;

[0056] The point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data;

[0057] The feature fusion network is used to fuse the 3D image features with the point cloud features of the voxels corresponding to the point cloud data to obtain fused features of the voxels of the first scene;

[0058] The output network is used to process the fused features of the voxels of the first scene and output the perception information.

[0059] In a possible implementation of the second aspect, the attribute recognition network includes a text encoding network and an attribute decoding network, wherein the text encoding network is used to extract word vector features of text query information; and the attribute decoding network is used to output category information of the obstacle based on the word vector features and the fusion features of the voxels of the obstacle.

[0060] In a possible implementation of the second aspect, the path evaluation network includes a path encoding network, a feature interaction network, and an evaluation output network, wherein:

[0061] The path encoding network is used to extract path features of each planned path in the multiple planned paths;

[0062] The feature interaction network is used to obtain the risk feature of each planned path according to the path feature of each planned path and the fusion feature of the voxels of the first scene;

[0063] The evaluation output network is used to output recommendation coefficients of the multiple planned paths and recommended paths among the multiple planned paths according to the risk characteristics of the multiple planned paths.

[0064] In a third aspect, the present application provides a device for intelligent driving, which includes: a receiving unit for obtaining data collected by a sensor for a first scene, wherein the sensor includes at least one of a camera and a radar; a processing unit for inputting the collected data into a perception detection network and outputting perception information, wherein the perception information is used to indicate voxels of obstacles in the first scene; the processing unit is also used to control vehicle driving based at least on the perception information.

[0065] In a possible implementation of the third aspect, the device further includes a display unit, which is used to display the obstacle based on the perception information, wherein the obstacle is marked with a polygonal box; and / or display voxels of the obstacle based on the perception information.

[0066] In a possible implementation of the third aspect, the perception information includes at least one item of the following information: the occupancy status of the voxels of the first scene, the speed information of the voxels of the first scene, the visibility status of the voxels of the first scene, and the corner point information of the polygonal box corresponding to the obstacle; wherein the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

[0067] In a possible implementation of the third aspect, the perception information is also used to indicate voxels of the road surface of the first scene, and the processing unit is specifically used to: generate road surface geometry information of the first scene based on at least the perception information; and adjust the suspension in the vehicle based on the road surface geometry information.

[0068] In a possible implementation manner of the third aspect, the processing unit is specifically used to: adjust the driving path of the vehicle at least based on the perception information, and the adjusted driving path does not pass through the area where the voxel of the obstacle is located.

[0069] In a possible implementation of the third aspect, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network, wherein:

[0070] The image feature extraction network is used to extract 3D image features of the image data;

[0071] The point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data;

[0072] The feature fusion network is used to fuse the 3D image features with the point cloud features of the voxels corresponding to the point cloud data to obtain fused features of the voxels of the first scene;

[0073] The output network is used to process the fused features of the voxels of the first scene and output the perception information.

[0074] In a possible implementation of the third aspect, the processing unit is further configured to: input text query information and fused features of the voxels of the obstacle into an attribute recognition network, and output category information of the obstacle; the text query information is used to request a query category; and the display unit is further configured to display the category information of the obstacle; wherein the fused features of the voxels of the obstacle are determined based on corner point information of a polygonal box corresponding to the obstacle and fused features of the voxels of the first scene, and the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

[0075] In a possible implementation of the third aspect, the receiving unit is further used to: obtain multiple planned paths for the vehicle; the processing unit is further used to: input the fusion features of the multiple planned paths of the vehicle and the voxels of the first scene into the path evaluation network, output the recommendation coefficients of the multiple planned paths and the recommended paths among the multiple planned paths, and the recommended paths are associated with the recommendation coefficients of the multiple planned paths; the display unit is further used to: display the recommended paths.

[0076] In a fourth aspect, the present application provides a device for intelligent driving, which includes a processor and a memory, wherein the memory is used to store program instructions; the processor calls the program instructions in the memory so that the device executes the method in the first aspect or any possible implementation of the first aspect.

[0077] In a fifth aspect, the present application provides a vehicle comprising a system as in the second aspect or any possible implementation of the second aspect, or comprising a device as in the third aspect or any possible implementation of the third aspect, or comprising a device as in the fourth aspect.

[0078] In a sixth aspect, the present application provides a computer-readable storage medium comprising computer instructions, which, when executed by a processor, implement the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0079] In a seventh aspect, the present application provides a computer program product that, when executed by a processor, implements the method described in the first aspect or any possible embodiment of the first aspect. The computer program product, for example, may be a software installation package. When the method provided in any possible design of the first aspect is required, the computer program product may be downloaded and executed on a processor to implement the method described in the first aspect or any possible embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] FIG1 is a schematic diagram of a communication system provided in an embodiment of the present application;

[0081] FIG2 is a system diagram of a perception model for intelligent driving provided in an embodiment of the present application;

[0082] FIG3 is a schematic diagram of feature extraction of a perception detection network provided in an embodiment of the present application;

[0083] FIG4 is a flow chart of an intelligent driving method provided by an embodiment of the present application;

[0084] FIG5 is a schematic diagram of some scenarios provided by embodiments of the present application;

[0085] FIG6A is a schematic diagram of marking obstacles in a scene with a polygonal frame according to an embodiment of the present application;

[0086] FIG6B is a schematic diagram showing a voxel display of an obstacle provided by an embodiment of the present application;

[0087] FIG6C is a schematic diagram showing voxels of a road surface in a scene provided by an embodiment of the present application;

[0088] FIG7A is a flowchart of a method for training a perception detection network according to an embodiment of the present application;

[0089] FIG7B is a schematic diagram of a training process of a perception detection network provided in an embodiment of the present application;

[0090] FIG8 is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0091] FIG9A is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0092] FIG9B is a schematic structural diagram of a training device provided in an embodiment of the present application;

[0093] FIG10 is a schematic structural diagram of a processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] It should be noted that the prefixes such as "first" and "second" used in this application are only for distinguishing different description objects, and do not have any limiting effect on the position, order, priority, quantity or content of the described objects. For example, if the described object is a "field", then the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the described object is a "level", then the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of described objects is not limited by the prefix and can be one or more. Taking "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the described object is a "device," then the "first device" and the "second device" can be the same device, the same type of device, or different types of devices. For another example, if the described object is "information," then the "first information" and the "second information" can be information of the same content or information of different contents. In short, the use of prefixes to distinguish the described objects in the embodiments of this application does not constitute a limitation on the described objects. For the description of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0095] It should be noted that the descriptions used in the embodiments of the present application, such as "at least one of a1, a2, ..., and an" and the like, include any one of a1, a2, ..., and an existing alone, and any combination of any multiple of a1, a2, ..., and an, each of which can exist alone. For example, the description "at least one of a, b, and c" includes a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a combination of ab and c.

[0096] To facilitate understanding, the following first introduces relevant terms that may be involved in the embodiments of this application.

[0097] (1) Autonomous driving

[0098] Autonomous driving, also known as intelligent driving or assisted driving, is a key development in the intelligent development of vehicles. With the advancement of perception technology and the improvement of chip capabilities, intelligent driving provides people with a growing number of rich driving features, gradually achieving different levels of driving experience. The Society of Automotive Engineers (SAE) has established a driving automation grading standard, which includes driving levels L0 to L5. Level L0 represents no automation, with the human driver fully in control of the vehicle, but receiving warnings or assistance from the driving system during driving, such as autonomous emergency braking (AEB), blind spot monitoring (BSM), or lane departure warning (LDW). Level 1 is driving assistance, in which the driving operation is completed by the human driver and the driving system. The driving system can provide driving assistance for the steering wheel or acceleration and deceleration operations through the driving environment, and other driving operations are performed by the human driver, such as adaptive cruise control (ACC) or lane keep assistance / support (LKA / LKS). Level 2 is partial automation, in which the driving environment provides driving assistance for the steering wheel and multiple acceleration and deceleration operations, and other driving actions are performed by the human driver, such as a combination of adaptive cruise control (ACC) and lane keep assistance (LKA / LKS). Level 3 is conditional automation, where the driving system can perform all driving operations, but the human driver needs to respond to the driving system's requests at appropriate times, meaning the human driver needs to be prepared to take over. Level 4 is highly automated, where the driving system can perform all driving operations, but the human driver does not necessarily need to respond to the driving system's requests. For example, when road and environmental conditions permit (such as closed campuses, highways, urban roads, or fixed driving routes), the human driver may not take over. Level 5 is fully automated, where the driving system can autonomously perform all driving operations under all road and environmental conditions that a human driver can handle. As can be seen, at levels 0 to 2, the driving system primarily provides support to the driver, but the driver still needs to supervise the driving and steer, brake, or accelerate as needed to ensure safety. At levels 3 to 5, the driving system can perform all driving operations on behalf of the driver. At level 3, the driver needs to be prepared to take over. At levels 4 and 5, the driving system can achieve full driving under some or all conditions, with the driver being able to choose whether to take over.

[0099] The above classifications are examples and may change with technological evolution or due to different regulations in different countries or regions. For example, the vehicle automation classification system proposed by the Ministry of Industry and Information Technology of China includes six levels of driving automation. Levels 0-2 are driving assistance, where the system assists humans in performing dynamic driving tasks, while the driver remains the primary driver. Levels 3-5 are automated driving, where the system replaces humans in performing dynamic driving tasks under designed operating conditions, but when the function is activated, the system is the primary driver. The names and definitions of the levels are as follows: Level 0 driving automation (emergency assistance) means the system cannot continuously perform lateral or longitudinal vehicle motion control during dynamic driving tasks, but has the ability to continuously detect and respond to some objects and events during dynamic driving tasks. Level 1 driving automation (partial driver assistance) means the system continuously performs lateral or longitudinal vehicle motion control during dynamic driving tasks under its designed operating conditions (or design operating range (ODD)) and has the ability to detect and respond to some objects and events appropriate to the lateral or longitudinal vehicle motion control being performed. Level 2 driving automation (combined driver assistance) systems continuously perform lateral and longitudinal vehicle motion control during dynamic driving tasks under their designed operating conditions and have the ability to detect and respond to certain objects and events that are appropriate to the lateral and longitudinal vehicle motion control being performed. Level 3 driving automation (conditionally automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions. Level 4 driving automation (highly automated driving) systems continuously perform all dynamic driving tasks under their designed operating conditions and automatically implement a minimal risk strategy. Level 5 driving automation (fully automated driving) systems continuously perform all dynamic driving tasks under all drivable conditions and automatically implement a minimal risk strategy. Lateral control primarily controls vehicle steering, for example, by controlling steering wheel torque or angle to steer the vehicle. Longitudinal control primarily controls vehicle speed, for example, by controlling the brake pedal, accelerator pedal, or gear position to control vehicle acceleration / deceleration and braking.

[0100] Regardless of the classification method adopted, the description of the embodiments of the present application can be applied to the above-mentioned automatic driving systems that require partial or full participation in vehicle driving.

[0101] (2) Obstacles

[0102] In the embodiments of the present application, an obstacle refers to an entity that the vehicle does not expect to collide with during driving. The entity can be static or dynamic. Among them, static entities can be, for example, cardboard boxes on the road, road construction signs, road dividing railings, piles of earth, tires, overturned vehicles, lying people, animals, buildings beside the road, trees, parked vehicles, road signs, telephone poles, roadside medians, and other stationary objects with volume and mass; dynamic entities can be, for example, pedestrians (such as walking pedestrians, cyclists, etc.), animals, vehicles, vehicles carrying goods (such as loaded with cardboard boxes, branches or other goods), and other moving objects with volume and mass.

[0103] (3) Scenario

[0104] A scene refers to the environmental space that can be detected by sensors on a vehicle while the vehicle is driving. It is understood that each moment in the vehicle's driving process corresponds to a scene, and scenes corresponding to multiple moments include scenes corresponding to each of the multiple moments.

[0105] (4) Voxel

[0106] A voxel, also known as a stereo pixel or volume element, is the smallest unit of segmentation in three-dimensional space, similar to the smallest unit of pixel in two-dimensional space. Voxels allow for the gridding of 3D space and the characterization of each grid. In this case, a voxel represents a value on a regular grid in three-dimensional space, and its position can be inferred based on its position relative to other voxels.

[0107] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0108] Referring to Figure 1 , which is a schematic diagram of a communication system provided in an embodiment of the present application, the system includes a network-side device and a vehicle, wherein the network-side device and the vehicle communicate with each other wirelessly.

[0109] Here, the network side device is a device with computing capabilities. The network side device can be, for example, a server deployed on the network side (such as a server for intelligent driving processing), or a component or chip in the server. In some possible embodiments, the network side device can also be a system-level device or a computing device cluster composed of multiple servers. The network side device can be deployed in a cloud environment or an edge environment, and the embodiments of this application are not specifically limited.

[0110] Here, a vehicle is one equipped with an autonomous driving system. Autonomous driving systems are not limited to fully autonomous driving systems, highly autonomous driving systems, conditionally autonomous driving systems, or partially autonomous driving systems. Those skilled in the art will appreciate that any non-fully manual driving system that provides intelligent driving is encompassed by this concept.

[0111] For example, depending on the power source of the vehicle, the vehicle may be a new energy vehicle or a traditional vehicle, etc., where a traditional vehicle refers to a fuel vehicle, such as a gasoline vehicle, a diesel vehicle, etc., and a new energy vehicle may be an electric vehicle (EV), a hybrid electric vehicle (HEV), an extended-range electric vehicle (range extended EV), a plug-in hybrid vehicle (Plug-in HEV), a fuel cell vehicle or other new energy vehicles, which are not specifically limited here.

[0112] The vehicle is equipped with cameras and radars, wherein the cameras are used to collect image data of the vehicle's current surroundings, and the radars are used to collect point cloud data of the vehicle's current surroundings. The radars include at least one of a laser radar (Lidar), a millimeter-wave radar (Radar), and the like. Based on the camera's installation position on the vehicle, the cameras can be divided into, for example, front-view cameras, surround-view cameras, rear-view cameras, and side-view cameras; based on the camera's structural division, the cameras can be divided into, for example, monocular cameras, binocular cameras, wide-angle cameras, and the like. Here, the embodiments of the present application do not limit the number of cameras configured on the vehicle. For safety reasons, the cameras on the vehicle need to be able to collect 360-degree image data around the vehicle body.

[0113] For example, a perception model is deployed on a network-side device, and the network-side device uses training data to train the perception model. The training data includes sensor data obtained from a data source device (e.g., a vehicle fleet), and the sensor data includes image data collected by an on-board camera and point cloud data collected by an on-board radar. After the network-side device has trained the perception model, it can provide the trained perception model to the vehicle for use. The specific training process of the perception model can be referred to the description of the corresponding content in the following method embodiment, and will not be repeated here.

[0114] Furthermore, the vehicle can obtain a perception model (i.e., a trained perception model) from a network-side device. During the vehicle's driving process, the vehicle uses its own sensors (such as cameras, radars, etc.) to collect data about the environment (or scene) within a certain range of the vehicle to obtain collected data. The collected data includes, for example, image data and point cloud data collected for the scene. The vehicle uses the perception model to process the collected data to output perception information of the scene. The perception information is used to indicate the voxels of obstacles in the scene. The vehicle can control its own driving based on at least this perception information.

[0115] The specific details of the perception model can be found in the relevant description of the embodiment of Figure 2 below, and will not be repeated here.

[0116] In the system shown in FIG1 , communication between the network-side device and the vehicle may utilize cellular communication technology, such as 2G cellular communication, such as the Global System for Mobile Communications (GSM) and General Packet Radio Service (GPRS); or 3G cellular communication, such as Wideband Code Division Multiple Access (WCDMA), Time Division-Synchronous Code Division Multiple Access (TS-SCDMA), Code Division Multiple Access (CDMA); or 4G cellular communication, such as Long Term Evolution (LTE), LTE-Vehicle to Everything (V2X) wireless communication technology, PC5 communication; or 5G cellular communication, such as New Radio (NR)-V2X PC5 communication, or other evolved cellular communication technologies. The wireless communication system may also utilize non-cellular communication technologies, such as Wi-Fi and Wireless Local Area Network (WLAN) communication, which are not specifically limited here.

[0117] It should be understood that Figure 1 is merely an exemplary architecture diagram and does not limit the number of network elements included in the system shown in Figure 1. Although not shown in Figure 1, Figure 1 may also include other functional entities in addition to the functional entities shown in Figure 1. Furthermore, the methods provided in the embodiments of the present application can be applied to the communication system shown in Figure 1. Of course, the methods provided in the embodiments of the present application can also be applied to other communication systems, and the embodiments of the present application are not limited in this regard.

[0118] See Figure 2, which is a system schematic diagram of a perception model for intelligent driving provided in an embodiment of the present application.

[0119] In Figure 2, the perception model includes a perception detection network. The perception detection network is used to output perception information based on sensor data collected from the scene (e.g., image data and point cloud data). The perception information is used to indicate voxels of obstacles in the scene. In some possible embodiments, the perception information is also used to indicate voxels of the road surface within the scene. The perception information can be used to assist vehicle driving.

[0120] The following introduces the framework of the perception detection network.

[0121] In one implementation, when the collected data includes image data and point cloud data, the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network and an output network, wherein the image feature extraction network is used to extract 3D image features of the image data from the image data and output the features to the feature fusion network, the point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data from the point cloud data and output the features to the feature fusion network, the feature fusion network is used to fuse the 3D image features of the image data and the point cloud features of the voxels corresponding to the point cloud data, obtain the fused features of the voxels of the corresponding scene and output the features to the output network, and the output network makes predictions based on the fused features of the voxels of the corresponding scene and outputs the perception information of the scene.

[0122] For example, the feature fusion network in Figure 2 can only perform spatial feature fusion. For example, if the image data is an image captured by a camera at time t, and the point cloud data is data collected by a radar at time t, then the feature fusion network only needs to perform spatial fusion on the 3D image features of the image data at time t and the point cloud features of the voxels corresponding to the point cloud data at time t.

[0123] For example, the feature fusion network in Figure 2 can perform spatial and temporal feature fusion. For example, if the image data is image data collected by a camera at n moments, and the point cloud data is point cloud data collected by a radar at these n moments, the feature fusion network can first spatially fuse the 3D image features corresponding to each of the n moments with the point cloud features of the voxels corresponding to that moment to obtain the spatial fusion features of the voxels corresponding to each moment, and then temporally fuse the spatial fusion features of the voxels corresponding to each of the n moments.

[0124] In some possible embodiments, if the collected data only includes image data, the point cloud feature extraction module in the perception detection network shown in Figure 2 can be omitted. If the image data is image data collected by the camera at n moments, the feature fusion network can temporally fuse the 3D image features corresponding to each of the n moments.

[0125] In some possible embodiments, if the collected data only includes point cloud data, the image feature extraction module in the perception detection network shown in Figure 2 can be omitted. If the point cloud data is point cloud data collected by the radar at n moments, the feature fusion network can temporally fuse the point cloud features of the voxels corresponding to each of the n moments.

[0126] Exemplarily, the feature fusion network can adopt a network structure of a recurrent neural network (RNN) or a recurrent convolutional neural network (RCNN). For example, a CNN can be a long short-term memory network (LSTM), a gated recurrent unit network (GRU), etc.

[0127] Furthermore, the image feature extraction network includes a camera backbone network and a stereo conversion network. The camera backbone network is used to extract 2D image features from image data, and the stereo conversion network is used to convert the 2D image features of the image data into 3D image features of the image data. Here, the stereo conversion network can convert 2D image features into 3D image features in the vehicle coordinate system. The features extracted from the radar point cloud data are themselves 3D features in the vehicle coordinate system. This facilitates the subsequent feature fusion network to fuse features from different sensors, which helps to eliminate heterogeneous differences between multimodal sensors.

[0128] Exemplarily, the 2D image features of the image data include, but are not limited to, color features, shape features, texture features, and spatial relationship features of the image data.

[0129] For example, the camera backbone network can adopt a convolutional neural network (CNN) (e.g., a residual network (ResNet), a transformer network, a vision transformer (ViT) network, or other backbone network structures. The stereo conversion network can adopt a transformer network or a lift-splat-shoot (LSS) network structure.

[0130] Furthermore, the point cloud feature extraction network includes a radar encoding network and a point backbone network, wherein the radar encoding network is used to voxelize the point cloud data to establish a correspondence between points and voxels in the point cloud data, thereby obtaining the features of the voxels corresponding to the point cloud data, and the point backbone network is used to extract the point cloud features (i.e., 3D features) of the voxels corresponding to the point cloud data based on the features of the voxels corresponding to the point cloud data. In some possible embodiments, the radar encoding network and the point backbone network can be merged into one network to extract the point cloud features of the voxels corresponding to the point cloud data, which is not specifically limited here. In some possible embodiments, if the computing power supports it, the radar encoding network and the point backbone network can also be merged into one network, which is used to extract the point cloud features of the voxels corresponding to the point cloud data.

[0131] For example, the radar encoding network may adopt a network structure such as a voxel feature encoding (VFE) network or a pillar feature encoding (PFE) network. The point backbone network may adopt a network structure such as a convolutional neural network (e.g., U-Net) or a transformer network.

[0132] The output network is the detection head of the perception detection network. The output network includes at least one head network, and the number of head networks in the output network is determined based on the number of types of prediction results in the perception information output by the output network. For example, as shown in Figure 2, the perception information includes the occupancy state of the voxel, the velocity information of the voxel, the visibility state of the voxel, and the corner point information of the polygonal box corresponding to the obstacle, wherein the polygonal box corresponding to the obstacle is associated with the voxel of the obstacle. It can be seen that the perception information contains 4 types of prediction results, so the output network includes four head networks, namely head network 1, head network 2, head network 3 and head network 4, wherein head network 1 is used to output the corner point information of the polygonal box corresponding to the obstacle, head network 2 is used to output the occupancy state of the voxel, head network 3 is used to output the velocity information of the voxel, and head network 4 is used to output the visibility state of the voxel.

[0133] Here, the visible state of a voxel means: in the scene where the vehicle is located at the current moment, if a voxel in the scene is not touched by the observation signal of any sensor (including camera and radar) on the vehicle at the current moment, then the visible state of the voxel is invisible; if the voxel is touched by the observation signal of at least one sensor, then the visible state of the voxel is visible.

[0134] For example, the occupancy state of a voxel in the scene currently occupied by the vehicle refers to: if a certain voxel in the scene has a physical entity at the corresponding spatial location in the physical world of the scene, then the occupancy state of the voxel is occupied; if no physical entity exists at the corresponding spatial location in the physical world of the scene, then the occupancy state of the voxel is empty (i.e., unoccupied). Here, a physical entity can be understood as an object with a certain volume and mass. It is understood that air is not a physical entity.

[0135] For example, any head network in the output network can adopt a convolutional neural network (CNN) or a transformer network structure. Here, the internal network structures of different head networks in the output network can be the same or different. It is understood that different head networks process the same input features differently.

[0136] In some possible embodiments, in order to reduce the consumption of computing power, in the perception detection network shown in FIG2 , a neural sampling network can also be set between the feature fusion network and the output network, that is, the feature fusion network outputs the fusion features of the voxels of the scene to the neural sampling network, and the neural sampling network processes the fusion features of the voxels of the scene at different resolutions according to the importance of the region in which the voxels of the scene are located. For example, if the importance of region one in the scene is greater than the importance of region two in the scene, the fusion features of the voxels in region one are processed at a first resolution, and the fusion features of the voxels in region two are processed at a second resolution, wherein the first resolution is greater than the second resolution. In this way, the neural sampling network can realize fine-grained processing of voxels in key areas of the scene and coarse-grained processing of voxels in non-key areas of the scene. This can greatly save computing power, improve the data processing efficiency of the perception detection network, and reduce the deployment cost of hardware.

[0137] For example, when the area 1 and the area 2 satisfy at least one of the following conditions, the importance of the area 1 is greater than the importance of the area 2:

[0138] (1) The distance from area 1 to the vehicle is smaller than the distance from area 2 to the vehicle;

[0139] (2) The number of obstacles in area 1 is greater than the number of obstacles in area 2;

[0140] (3) The number of dynamic obstacles in area one is greater than the number of dynamic obstacles in area two; and

[0141] (4) The volume of the obstacle in area 1 is greater than the volume of the obstacle in area 2.

[0142] For example, the neural sampling network may adopt a network structure of a neural network, a multi-layer perceptron (MLP), or a transformer network.

[0143] In order to more clearly show the process of feature extraction of the perception detection network, refer to Figure 3, which is a schematic diagram of feature extraction of a perception detection network provided by an embodiment of the present application. In Figure 3, the image data collected by n cameras can be used to extract the 2D image features of the image data through the above-mentioned camera backbone network, and the 2D image features of the image data can be used to extract the 3D image features of the image data through the stereo conversion network. The point cloud data collected by the radar can be used to extract the features of the voxels corresponding to the point cloud data (i.e., 3D features) through the radar encoding network. The features of the voxels corresponding to the point cloud data can be used to extract the point cloud features of the voxels corresponding to the point cloud data (i.e., 3D features) through the point backbone network. The 3D image features of the above-mentioned image data and the point cloud features of the voxels corresponding to the point cloud data are fused through the feature fusion network to output the fusion features of the voxels. Finally, the fusion features of the voxels are output through the above-mentioned output network to output the occupancy status of the voxels, the speed information of the voxels, the visible status of the voxels, and the corner point information of the polygonal box corresponding to the obstacle.

[0144] It can be understood that FIG3 is only an example of the feature extraction process of the perception detection network, and does not limit the feature extraction process in the perception detection network to that shown in FIG3 .

[0145] In some possible embodiments, the perception model also includes an attribute recognition network, which can be used to identify the category of the obstacle. Exemplarily, the attribute recognition network is used to output the category information of the obstacle based on the text query information and the fusion features of the voxels of the obstacle, wherein the text query information is used to request the query category, and the fusion features of the voxels of the obstacle are determined based on the corner point information of the polygonal box corresponding to the obstacle and the fusion features of the voxels of the scene, and the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle. As can be seen from Figure 2, the corner point information of the polygonal box corresponding to the obstacle comes from the output network in the perception detection network (specifically, the head network 1 in the output network), and the fusion features of the voxels of the scene are the output of the feature fusion network in the perception detection network.

[0146] Here, the association between the polygonal box corresponding to the obstacle and the voxels of the obstacle can be understood as follows: the corner point information of the polygonal box corresponding to the obstacle is obtained based on the index information of the voxels of the obstacle. The corner point information of the polygonal box corresponding to the obstacle can be obtained, for example, by predicting the index information of the voxels of the obstacle based on the learned rules of the head network 1 in Figure 2, or by calculating the corner point information of the polygonal box corresponding to the obstacle based on the index information of the voxels of the obstacle using a convex hull algorithm, without specific limitation here.

[0147] Here, the polygonal frame corresponding to the obstacle can be two-dimensional or three-dimensional, and is not specifically limited here.

[0148] In one implementation, the attribute recognition network includes a text encoding network and an attribute decoding network, wherein the text encoding network is used to extract word vector features of text query information; the attribute decoding network is used to output the category information of the obstacle based on the fusion features of the word vector features and the voxels of the obstacle.

[0149] Here, the text query information is used to request a query of Q categories. Assume that the number of obstacle categories in a scene is P, where Q and P are positive integers and Q is greater than P. In other words, the number of categories that the attribute recognition network actually supports is greater than the number of obstacle categories in any scene. This ensures that the attribute recognition network avoids omissions when identifying obstacle categories in any scene.

[0150] For example, in the category reasoning process, the text query information includes K pieces of text query information, such as "Is it a car?", "Is it a pedestrian?", "Is it a telephone pole?", "Is it a road sign?", "Is it a road dividing railing?", etc. The text encoding network in the attribute recognition network performs feature extraction on the K pieces of text query information to obtain the word vector features corresponding to each piece of text query information, wherein the word vector features corresponding to each piece of text query information can represent the image semantic features of the category indicated by the text query information. Taking the attribute decoding network in the attribute recognition network for the type identification of obstacle 1 as an example, obstacle 1 is any obstacle in the scene. The attribute decoding network calculates the similarity between the fused features of the voxels of obstacle 1 and the word vector features corresponding to each piece of text query information in the K pieces of text query information, and determines that the category corresponding to the word vector feature with the highest similarity to the fused features of the voxels of obstacle 1 is the category of obstacle 1, so that the category information of obstacle 1 can be output.

[0151] Here, the fusion features of the voxels of obstacle 1 can be determined based on the corner point information of the polygonal box corresponding to obstacle 1 and the fusion features of the voxels of the scene: since the corner point information of the polygonal box corresponding to obstacle 1 corresponds to the index information of the voxels of obstacle 1, the fusion features of the voxels of obstacle 1 can be determined from the fusion features of the voxels of the scene based on the index information of the voxels of obstacle 1.

[0152] For example, the text encoding network and the attribute decoding network can both adopt the network structure of a convolutional neural network or a transformer network. It is understood that the text encoding network and the attribute decoding network can adaptively adjust the relevant parameters of the network according to their own functions.

[0153] In some possible embodiments, the perception model further includes a path evaluation network, which can be used to determine a recommended path for the vehicle. Exemplarily, the path evaluation network is used to output recommendation coefficients for the multiple planned paths and a recommended path among the multiple planned paths based on fused voxel features of the vehicle and the scene.

[0154] In one implementation, the path evaluation network includes a path encoding network, a feature interaction network, and an evaluation output network, wherein the path encoding network is used to extract the path features of each planned path among multiple planned paths of the vehicle; the feature interaction network is used to obtain the risk features of each planned path based on the path features of each planned path and the fusion features of the voxels of the scene; and the evaluation output network is used to output the recommendation coefficients of the multiple planned paths and the recommended paths among the multiple planned paths based on the risk features of the multiple planned paths.

[0155] Exemplarily, the recommendation coefficient of the planned path can be obtained based on at least one of the risk coefficient, comfort and traffic efficiency of the planned path. Among them, the risk coefficient of the planned path is related to at least one of the factors such as the distance between the planned path and obstacles (including visible obstacles and obstacles currently in blind spots), whether the planned path conflicts with the paths of other traffic participants on the road (for example, whether a collision will occur at the current moment or in the future), etc. The traffic efficiency of the planned path is related to at least one of the factors such as the length of the planned path, the estimated travel time corresponding to the planned path, the number of traffic lights along the planned path, and the area of ​​the drivable area where the planned path is located. The comfort of the planned path is related to at least one of the factors such as the magnitude and frequency of the steering acceleration of the planned path, the rate of change of the acceleration of the planned path, the flatness of the road surface along the planned path, the number of traffic lights along the planned path, the type of road where the planned path is located, and whether the area along the planned path is shady.

[0156] Exemplarily, the recommended path is the planned path corresponding to the highest recommendation coefficient among the multiple planned paths.

[0157] For example, the path encoding network can adopt a network structure of a convolutional neural network, a transformer network, a graph neural network (GNN), or a graph convolution neural network (GCNNs). The feature interaction network can adopt a network structure of a graph neural network or a transformer network. The evaluation output network can adopt a network structure of a neural network or a multilayer perceptron (MLP).

[0158] It can be understood that the framework of the perception model shown in Figure 2 is only a feasible example given in the embodiment of this application and should not constitute a limitation on the framework of the perception model.

[0159] For example, when the perception model includes a perception detection network, an attribute recognition network, and a path evaluation network, the perception detection network, the attribute recognition network, and the path evaluation network can be trained separately. For example, the perception detection network is trained first, and after the perception detection network training is completed, the attribute recognition network and the path evaluation network are trained in sequence. The perception detection network, the attribute recognition network, and the path evaluation network can also be trained simultaneously, which is not specifically limited here. The training process of each network in the perception model can be referred to the description of the corresponding content in the following embodiments and will not be repeated here.

[0160] See Figure 4, which is a flow chart of an intelligent driving method provided by an embodiment of the present application. The method can be applied to the vehicle in Figure 1 above or a component (such as a chip or integrated circuit, etc.) used for autonomous driving control on the vehicle, where at least the above-mentioned perception detection network is deployed. The method includes but is not limited to the following steps:

[0161] S401: Acquire data collected by a sensor on a first scene, where the sensor includes at least one of a camera and a radar.

[0162] Here, the first scene can be understood as the environmental space that can be detected by the sensor while the vehicle is driving.

[0163] Here, sensors are deployed on the vehicle. Based on their installation locations on the vehicle, cameras can be categorized as front-view cameras, surround-view cameras, rear-view cameras, and side-view cameras, for example. Radars include at least one of lidar and millimeter-wave radars. The embodiments of this application do not limit the number of cameras or radars deployed on the vehicle.

[0164] Among them, the camera is used to collect image data, and the radar is used to collect point cloud data, so the above-mentioned collected data includes at least one of image data and point cloud data.

[0165] For example, a vehicle can be equipped with multiple cameras, each with a different field of view. These cameras can cover a 360-degree field of view centered on the vehicle. For example, the fields of view of adjacent cameras can partially overlap. This allows data from the same environment to be collected simultaneously by multiple sensors, improving the confidence of the data observations.

[0166] Exemplarily, the sensor includes a camera and a radar. Assume that the number of cameras on the vehicle is m, and the m cameras collect image data for the first scene. Assume that each camera collects an image at each moment, which means that the collected data corresponding to each moment includes the image data corresponding to the m images collected by the camera and the point cloud data collected by the radar.

[0167] S402: Input the collected data into a perception detection network and output perception information, where the perception information is used to indicate voxels of obstacles in the first scene.

[0168] Here, obstacles refer to entities that a vehicle is not expected to collide with during driving. Such entities can be static or dynamic. Static entities can include, for example, cardboard boxes on the road, road construction signs, road dividing railings, dirt piles, tires, overturned vehicles, lying people, animals, buildings beside the road, trees, parked vehicles, road signs, telephone poles, roadside medians, and other stationary objects with volume and mass. Dynamic entities can include, for example, pedestrians (e.g., walking pedestrians, cyclists, etc.), animals, vehicles, and vehicles carrying cargo (e.g., loaded with cardboard boxes, branches, or other cargo).

[0169] Here, the perception detection network is a pre-trained perception detection network deployed on the vehicle. The perception detection network is configured to output perception information based on sensor data collected from the first scene. For example, the perception detection network is trained by the network-side device shown in Figure 1 based on a sensor dataset and label information corresponding to the sensor dataset generated using 4D reconstruction. The label information corresponding to the sensor dataset can be generated by the network-side device using a self-supervised 4D reconstruction based on the sensor dataset. This label information is used to provide the perception detection network with true value information for its prediction results during training.

[0170] For example, the prediction tasks of the perception detection network include predicting the occupancy status of voxels, the velocity information of voxels, the visibility status of voxels, and the corner point information of the polygonal box corresponding to the obstacle. During the training process of the perception detection network, assuming that the current input data of the perception detection network is the image data at time t and the point cloud data at time t, the perception detection network performs the above four prediction tasks on the input data and outputs the predicted perception information (i.e., the prediction result). Accordingly, the label information includes the true value information of the prediction results corresponding to the image data at time t and the point cloud data at time t.

[0171] In one implementation, the collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network. In this case, the processing of the perception detection network can refer to the following steps A1-A4, for example:

[0172] A1: The image feature extraction network extracts 3D image features of the image data;

[0173] A2: The point cloud feature extraction network extracts the point cloud features of the voxels corresponding to the point cloud data;

[0174] A3: The feature fusion network fuses the above-mentioned 3D image features and the point cloud features of the voxels corresponding to the point cloud data to obtain the fused features of the voxels of the first scene;

[0175] A4: The output network processes the fused features of the voxels of the first scene and outputs the perception information.

[0176] The reasoning process of the perception detection network is described in detail in the embodiment of FIG2 . The image feature extraction network, point cloud feature extraction network, feature fusion network, and output network are described in detail in the embodiment of FIG2 , and are not further described here. It should be understood that the above example does not limit the framework of the perception detection network.

[0177] In an embodiment of the present application, the perception information includes at least one of the following information: the occupancy status of the voxels of the first scene, the speed information of the voxels of the first scene, the visibility status of the voxels of the first scene, and the corner point information of the polygonal box corresponding to the obstacle of the first scene; wherein, the polygonal box corresponding to the obstacle of the first scene is associated with the voxels of the obstacle.

[0178] For example, referring to the framework of the perception detection network shown in Figure 2, it can be seen that the output network includes four head networks, each head network corresponds to a prediction task. In this case, the perception information includes the occupancy status of the voxels of the first scene, the velocity information of the voxels of the first scene, the visibility status of the voxels of the first scene, and the corner point information of the polygonal box corresponding to the obstacles of the first scene.

[0179] Here, the occupancy state of a voxel can be divided into two types, namely, “occupied” and “empty.” For details about the occupancy state of a voxel, please refer to the aforementioned description of the occupancy state of a voxel, which will not be repeated here.

[0180] For example, voxel 1 of the scene corresponds to vehicle A in the physical world where the scene is located, then the occupancy state of voxel 1 is "occupied"; voxel 2 of the scene corresponds to the air in the physical world where the scene is located, then the occupancy state of voxel 2 is "empty".

[0181] Here, the voxel can have two visible states, namely, “visible” and “invisible.” For details about the visible state of a voxel, please refer to the aforementioned description of the visible state of a voxel, which will not be repeated here.

[0182] Exemplarily, the visible state of a voxel can be changed. See Figure 5, which is a schematic diagram of some scenarios provided by an embodiment of the present application. (1) of Figure 5 shows scene 1 corresponding to time t1. In (1) of Figure 5, vehicle 1 is the main vehicle (i.e., the above-mentioned perception detection network is deployed on vehicle 1). It can be seen that vehicles 1, 2, and 3 are located in the same lane, and vehicle 2 is currently performing a lane change operation. Assuming that the vehicle size of vehicle 2 is larger than the vehicle size of vehicle 3 in front, vehicle 3 is completely blocked by vehicle 2 from the perspective of vehicle 1. Vehicle 3 is located in the blind spot of vehicle 1. Therefore, at time t1, the observation signal of any sensor on vehicle 1 cannot reach the voxel of vehicle 3. Therefore, in the perception information output by vehicle 1, the visible state of the voxel of vehicle 2 at time t1 is "visible", but the visible state of the voxel of vehicle 3 at time t1 is "invisible". Figure 5 (2) shows scene 2 corresponding to time t2. It can be seen that vehicle 2 has completed the lane change operation. Assuming that both vehicle 2 and vehicle 3 appear within the acquisition field of view of the sensor of vehicle 1, it means that the voxels of vehicle 2 and vehicle 3 can be touched by the observation signal of at least one sensor on vehicle 1 at time t2. Therefore, in the perception information output by vehicle 1, the visible state of the voxels of vehicle 2 at time t2 is "visible" and the visible state of the voxels of vehicle 3 at time t2 is also "visible". It can also be seen from this that inputting the collected data at multiple times into the perception detection network can not only supplement the observation information, but also help to restore the physical world where the scene is located more realistically from multiple directions and angles.

[0183] S403: Control vehicle driving based at least on the perception information.

[0184] Here, controlling the vehicle's movement includes at least one of the following: changing lanes, adjusting speed, adjusting the driving path, activating warning lights, and adjusting the vehicle's suspension. This allows the vehicle to make real-time decisions based on at least some of the perceived information, thereby improving driving safety.

[0185] In one implementation, controlling vehicle travel based at least on the perception information includes adjusting a vehicle travel path based at least on the perception information, wherein the adjusted travel path does not pass through an area where voxels of obstacles are located.

[0186] For example, based on the perception information, it can be determined whether the vehicle's current driving path will collide with the voxels of the obstacle at the corresponding moment in the scene at the current moment and in the future. If a collision is predicted, the vehicle's current driving path can be adjusted in time so that the adjusted driving path does not pass through the area where the obstacle voxels are located. In this way, the vehicle can be prevented from colliding with the obstacle during driving, which is conducive to improving the safety of vehicle driving.

[0187] In one implementation, the perception information is also used to indicate voxels of the road surface of the first scene, and the vehicle driving is controlled at least based on the perception information, including: generating road surface geometry information of the first scene based on at least the perception information; and adjusting the suspension in the vehicle based on the road surface geometry information.

[0188] Here, the road geometry information is used to indicate the road condition of the first scene (for example, whether there are potholes or bumps on the road, etc.). Based on the perception information, the vehicle can obtain the road condition in front of the vehicle in advance. When the road surface is detected to be undulating, the vehicle has enough time to adjust the vehicle's suspension in time to keep the vehicle as level and stable as possible during driving, reducing the vibration caused by road undulations and improving the comfort of riding in the vehicle.

[0189] Exemplarily, controlling vehicle travel based on perception information may also include: determining the blind spot in the scene and obstacle information within the blind spot (e.g., speed information of the obstacle, corner point information of the polygonal box opposite the obstacle, etc.) based on the perception information; when the vehicle approaches the blind spot, controlling the vehicle to decelerate, stop, or turn based on the obstacle information within the blind spot. Here, the type and form of obstacles within the blind spot are not limited, and obstacles within the blind spot may be static or dynamic, which are not specifically limited here. In this way, when the vehicle approaches the blind spot in the current scene, controlling the vehicle to be in a decelerating state, a stopping state, or a turning state can avoid collision between the vehicle and obstacles within the blind spot, thereby improving the safety of vehicle driving.

[0190] Here, a blind spot is, for example, the area within the sensor information where voxels with a current visibility status of "invisible" are located. For example, the blind spot includes areas where the observation signal from the vehicle's sensor could have reached but are blocked by obstacles, as well as the sensor's own detection blind spot.

[0191] Here, the speed information of the obstacle can be obtained based on the speed information of the voxels of the obstacle, for example.

[0192] In some possible embodiments, in addition to controlling vehicle driving based on the perception information output by the vehicle itself, the vehicle can also control vehicle driving by combining at least one of navigation map information, high-precision map information, live traffic information broadcast by roadside equipment, and live traffic information broadcast by other surrounding vehicles. Here, the roadside equipment can be, for example, a roadside unit (RSU), multi-access edge computing (MEC), or sensor, or a component or chip within these devices, or a system-level device consisting of an RSU and MEC, or a system-level device consisting of an RSU and a sensor, or a system-level device consisting of an RSU, MEC, and a sensor.

[0193] Optionally, in some possible embodiments, the above-mentioned intelligent driving method further includes: displaying obstacles in the first scene based on the perception information, wherein the obstacles in the first scene are marked with polygonal boxes; and / or displaying voxels of the obstacles in the first scene based on the perception information.

[0194] For example, obstacles or voxels of obstacles can be displayed on a display device of the vehicle. For example, the display device can be a vehicle-side tablet, an onboard display, a head-up display (HUD) system, or an augmented head-up display (AR-HUD) system, etc., without specific limitation herein.

[0195] Refer to Figure 6A, which is a schematic diagram of marking obstacles in a scene with polygonal boxes provided by an embodiment of the present application. Figure 6A shows the obstacles in the scene where the ego vehicle is currently located, wherein the obstacles are marked with polygonal boxes. The vehicle at the bottom center of Figure 6A is the ego vehicle. It can be seen that the obstacles in the environment surrounding the ego vehicle in this scene are marked and displayed with polygonal boxes. Based on the shape of the polygonal boxes, it can be seen that the obstacles in the scene include at least vehicles, buildings, etc. For example, the polygonal box can be two-dimensional or three-dimensional. Taking the polygonal box closest to the ego vehicle on the right side of Figure 6A as an example, when the polygonal box is displayed in 2D, the polygonal box can be connected by 10 corner points indicated by one set of corner point information; when the polygonal box is displayed in 3D, the polygonal box can be connected by multiple sets of corner point information, wherein each set of corner point information indicates 10 corner points. In some possible embodiments, for dynamic obstacles in the scene, an arrow may be added to the polygonal box corresponding to the dynamic obstacle. This arrow indicates that the obstacle is dynamic, and the direction of the arrow indicates the direction of movement of the obstacle. The length of the arrow indicates the speed of the obstacle. It should be understood that Figure 6A is only an example of the marking display of obstacles in the scene where the vehicle is located at a certain moment and should not constitute a limitation on the marking display of obstacles in the scene where the vehicle is located.

[0196] Refer to Figure 6B, which is a schematic diagram of the display of voxels of an obstacle provided in an embodiment of the present application. Figure 6B shows the voxels of the obstacle in the scene where the vehicle is currently located. It can be seen that the voxels of the obstacle are composed of multiple voxels in the scene, and the voxel can be understood as the three-dimensional grid of the smallest unit in Figure 6B. For example, in Figure 6B, dynamic obstacles and static obstacles can be distinguished and displayed by different colors (that is, obstacles of different speeds can be distinguished by different colors), and different obstacles can also be distinguished by different colors, which is not specifically limited here. It can be understood that Figure 6B is only an example of the display of voxels of obstacles in the scene where the vehicle is located at a certain moment, and should not limit the display of voxels of obstacles in the scene where the vehicle is located.

[0197] In some possible embodiments, the voxels of the road surface in the scene where the vehicle is currently located can also be displayed. Referring to FIG6C , FIG6C is a schematic diagram showing the voxels of the road surface in the scene provided by an embodiment of the present application. FIG6C not only displays the voxels of the obstacles in the scene at the current moment, but also displays the voxels of the road surface in the scene at the current moment. Thus, the degree of undulation of the road surface ahead can be seen based on FIG6C . It can be understood that FIG6C is only an example of displaying the voxels of the obstacles and the voxels of the road surface in the scene where the vehicle is located at a certain moment, and should not limit the display of the voxels of the obstacles and the voxels of the road surface in the scene where the vehicle is located.

[0198] In some possible embodiments, in addition to the aforementioned perception and detection network, a vehicle may also be equipped with an attribute recognition network, which is used to identify the type of obstacles. This allows the vehicle to not only detect obstacles in its surroundings but also identify their types, enabling it to not only see but also understand them.

[0199] Furthermore, the intelligent driving method further includes: obtaining text query information; inputting the text query information and the fusion features of the voxels of the obstacle into an attribute recognition network, and outputting the category information of the obstacle; wherein the text query information is used to request a query category; and displaying the category information of the obstacle; wherein the fusion features of the voxels of the obstacle are determined based on the corner point information of the polygonal box corresponding to the obstacle and the fusion features of the voxels of the first scene, and the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle. The corner point information of the polygonal box corresponding to the obstacle and the fusion features of the voxels of the first scene are both derived from the perception detection network. Furthermore, in conjunction with the perception detection network shown in FIG2 , it can be seen that the corner point information of the polygonal box corresponding to the obstacle comes from the output network of the perception detection network, and the fusion features of the voxels of the first scene come from the feature fusion network of the perception detection network. For details of this embodiment, please refer to the relevant description of the attribute recognition network in the embodiment of FIG2 . For the sake of brevity, it will not be repeated here.

[0200] In some possible embodiments, after obtaining the above-mentioned perception information, further fusion processing can be performed in combination with the detection results of the detection algorithm configured in the camera, the detection results of the detection algorithm configured in the radar, or the detection results of other models. In this way, when the same obstacle can be perceived in multiple different ways, the confidence level of detecting the obstacle will be higher.

[0201] In some possible embodiments, in addition to the aforementioned perception and detection network, a path assessment network can also be deployed on the vehicle. The path assessment network is used to recommend the lowest-risk path for the vehicle. This helps improve driving safety and the accuracy of driving decisions.

[0202] Furthermore, the above-mentioned intelligent driving method also includes: obtaining multiple planned paths for the vehicle; inputting the fusion features of the multiple planned paths of the vehicle and the voxels of the first scene into the path evaluation network, outputting the recommendation coefficients of the multiple planned paths and the recommended paths among the multiple planned paths, wherein the recommended paths are associated with the recommendation coefficients of the multiple planned paths; and displaying the recommended paths. It can be seen that the fusion features of the voxels of the first scene come from the perception detection network. Combined with the description of the perception detection network in Figure 2 above, it can be seen that the fusion features of the voxels of the first scene are provided by the feature fusion network in the perception detection network. Here, multiple planned paths are generated by the vehicle, for example, the vehicle generates multiple planned paths based on navigation map information. For this embodiment, please refer to the relevant description of the path evaluation network in the embodiment of Figure 2 above. For the sake of brevity of the specification, it will not be repeated here.

[0203] Exemplarily, the recommended path output by the path evaluation network includes at least two planned paths from a plurality of planned paths. In this case, in a human-machine co-driving scenario, it can also be used to recommend the recommended path to the user, receive feedback information from the user, and the feedback information is used to indicate the path selected by the user from the at least two planned paths, and control the vehicle itself to travel along the path selected by the user.

[0204] It can be understood that when the recommended path includes multiple planned paths, it can be understood that the recommendation coefficients of the planned paths included in the recommended path are similar or the same, but some planned paths are the shortest paths, some planned paths are the most comfortable paths, and some planned paths are the shortest paths, etc. In this case, users can freely choose according to their own needs, providing users with a good riding experience.

[0205] In some possible embodiments, the above-mentioned perception detection network, attribute recognition network and path evaluation network may also be deployed on the vehicle at the same time. For corresponding descriptions, please refer to the descriptions of the corresponding embodiments and will not be repeated here.

[0206] As can be seen, by implementing the embodiments of the present application, deploying the aforementioned perception detection network on a vehicle can enhance the vehicle's ability to perceive its surrounding environment, enabling it to sense surrounding obstacles while driving, thereby avoiding collisions and improving vehicle safety. Furthermore, by deploying the aforementioned attribute recognition network on a vehicle, the vehicle can not only sense obstacles but also identify their types, thereby enhancing the vehicle's intelligence.

[0207] See Figure 7A, which is a flow chart of a training method for a perception detection network provided in an embodiment of the present application. This method can be applied to the network-side device shown in Figure 1 above or components within the network-side device (such as chips or integrated circuits, etc.). Taking the perception detection network shown in Figure 2 as an example, the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network, and an output network, wherein the output network includes multiple head networks. The method includes but is not limited to the following steps:

[0208] S701: During each training process, feature extraction is performed on image data at each moment in a batch of sensor data through an image feature extraction network to obtain 3D image features of image data at K moments.

[0209] Illustratively, a batch of sensor data used in each training process is located in a sensor dataset.

[0210] For example, a batch of sensor data includes image data at K moments, where the image data at K moments come from at least one camera of the vehicle, and K is a positive integer.

[0211] Specifically, the image data of K moments in a batch of sensor data are input into the image feature extraction network. The image feature extraction network obtains the 3D image features of the image data at each moment based on the image data at that moment. Therefore, the image feature extraction network will obtain the 3D image features of the image data at K moments and output them to the feature fusion network.

[0212] Here, the framework of the image feature extraction network can be specifically referred to the description of the corresponding content in the embodiment of Figure 2, and will not be repeated here.

[0213] S702: During each training process, feature extraction is performed on the point cloud data at each moment in a batch of sensor data through a point cloud feature extraction network to obtain point cloud features of voxels corresponding to the point cloud data at K moments.

[0214] Here, a batch of sensor data also includes point cloud data at these K moments, wherein the point cloud data at these K moments come from at least one radar of the vehicle.

[0215] Specifically, the point cloud data of K moments in a batch of sensor data are input into the point cloud feature extraction network. The point cloud feature extraction network obtains the point cloud features of the voxels corresponding to the point cloud data at each moment based on the point cloud data at that moment. Therefore, the point cloud feature extraction network will obtain the point cloud features of the voxels corresponding to the point cloud data at K moments and output them to the feature fusion network.

[0216] S703: Perform feature fusion on the 3D image features of the image data at K moments and the point cloud features of the voxels corresponding to the point cloud data at K moments through a feature fusion network to obtain fused features of the voxels of the scene at K moments.

[0217] Here, the voxels of the scene refer to the voxels after feature fusion is performed by the feature fusion network.

[0218] Exemplarily, the feature fusion network spatially fuses the 3D image features of the image data at each moment and the point cloud features of the voxels corresponding to the point cloud data at that moment to obtain the spatial fusion features of the voxels of the scene at that moment; the feature fusion network then temporally fuses the spatial fusion features of the voxels of the scene at K moments to obtain the fusion features of the voxels of the scene at K moments. In this case, the fusion features of the voxels of the scene at each moment can be called the spatiotemporal fusion features of the voxels of the scene at that moment.

[0219] S704: Outputting K prediction results according to the fusion features of the voxels of the scenes at K moments through each head network in the output network, wherein each prediction result of the K prediction results corresponds to a scene at one moment.

[0220] Among them, the fused features of the voxels of the scene at K moments correspond to each head network in the output network, and each head network in the output network performs a prediction task.

[0221] For example, the output network shown in Figure 2 includes head network 1, head network 2, head network 3 and head network 4, where head network 1 is used to predict the corner point information of the polygonal box corresponding to the obstacle in the scene, head network 2 is used to predict the occupancy state of the voxel in the scene, head network 3 is used to predict the velocity information of the voxel in the scene, and head network 4 is used to predict the visibility state of the voxel in the scene.

[0222] Specifically, in the output network, each head network makes a prediction based on the fused features of the voxels of the scene at each moment, and obtains the prediction result corresponding to the scene at that moment, so that each head network can obtain K prediction results.

[0223] Taking the head network 1 in the output network shown in Figure 2 as an example, it is assumed that K moments include t1, t2, ..., t K At time t1, head network 1 outputs prediction result 1 based on the fusion features of the voxels of the scene at time t1, and prediction result 1 includes the corner point information of the polygonal box corresponding to the obstacle in the scene at time t1; head network 1 outputs prediction result 2 based on the fusion features of the voxels of the scene at time t2, and prediction result 2 includes the corner point information of the polygonal box corresponding to the obstacle in the scene at time t2; ..., in this way, head network 1 will output K prediction results based on the fusion features of the voxels of the scenes at K times.

[0224] S705: Obtain the loss value of each head network in the output network according to the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network.

[0225] The label information corresponding to the batch of sensor data is generated based on the batch of sensor data through 4D reconstruction. The label information corresponding to the batch of sensor data is used to provide the perception detection network with the true value information corresponding to the perception detection network's prediction result for the batch of sensor data.

[0226] Taking the output network shown in Figure 2 as an example, the label information corresponding to this batch of sensor data includes the true value information of the corner points of the polygonal box corresponding to the obstacles in the scene at K moments, the true value information of the occupancy state of the voxels in the scene at these K moments, the true value information of the velocity of the voxels in the scene at these K moments, and the true value information of the visible state of the voxels in the scene at these K moments, wherein the true value information of the corner points of the polygonal box corresponding to the obstacles in the scene at K moments corresponds to the K prediction results output by the above-mentioned head network 1, the true value information of the occupancy state of the voxels in the scene at these K moments corresponds to the K prediction results output by the above-mentioned head network 2, the true value information of the velocity of the voxels in the scene at these K moments corresponds to the K prediction results output by the above-mentioned head network 3, and the true value information of the visible state of the voxels in the scene at these K moments corresponds to the K prediction results output by the above-mentioned head network 4.

[0227] Taking the calculation of the loss value of head network 1 as an example, the loss value of head network 1 is obtained based on the ground-truth information of the corner points of the polygonal box corresponding to the obstacles in the scene at K moments in time in the aforementioned label information and the K prediction results output by head network 1. For example, the loss value of head network 1 at each moment can be first obtained based on the ground-truth information of the corner points of the polygonal box corresponding to the obstacles in the scene at that moment and the prediction result at that moment among the K prediction results output by head network 1 (i.e., including the corner point information of the polygonal box corresponding to the obstacles in the scene at that moment). The loss value of head network 1 can then be obtained based on the loss value of head network 1 at each of the K moments in time. Similarly, other head networks in the output network can also use this method to calculate their own loss values. In this way, the loss value of each head network in the output network can be obtained.

[0228] S706: Weight the loss values ​​of each head network in the output network to obtain a loss value corresponding to each training process; and use the loss value to update the parameters in the perception detection network.

[0229] Here, the weight of each head network in the output network can be user-defined.

[0230] After obtaining a loss value corresponding to each training process, the loss value is used to update the parameters in the perception detection network (for example, each head network + feature fusion network + image feature extraction network + point cloud feature extraction network in the output network).

[0231] It is understood that Figure 7A is an example of training the above-mentioned perception detection network alone, and does not limit the training process of the perception detection network to the form shown in Figure 7A. In some possible embodiments, each head network in the output network of the above-mentioned perception detection network can also be trained separately. In this case, the above-mentioned S706 is not a necessary execution step. In some possible embodiments, the perception detection network can also be jointly trained with the neural radiation field NeRF network to further improve the detection accuracy and training efficiency, which is not specifically limited here.

[0232] By implementing the embodiments of the present application, the label information corresponding to a batch of sensor data during each training process does not need to be manually annotated, which not only saves manpower but also improves the efficiency of label information acquisition. The perception detection network uses self-supervised training to learn how to accurately predict the perception information of the scene in which the vehicle is located at any moment based on the input data of the vehicle at that moment.

[0233] See Figure 7B, which is a schematic diagram of the training process of a perception detection network provided in an embodiment of the present application.

[0234] As shown in Figure 7B, a batch of sensor data in the sensor data set is input into the image feature extraction network and the point cloud feature extraction network in the perception detection network. The image feature extraction network performs feature extraction on the image data in the batch of sensor data to obtain the 3D image features of the image data. The point cloud feature extraction network performs feature extraction on the point cloud data in the batch of sensor data to obtain the point cloud features of the voxels corresponding to the point cloud data. The feature fusion network in the perception detection network performs feature fusion on the 3D image features of the image data from the image feature extraction network and the point cloud features of the voxels corresponding to the point cloud data from the point cloud feature extraction network to obtain the fused features of the voxels of the scene and input them into the output network in the perception detection network. Each head network in the output network obtains the corresponding prediction results based on the fused features of the voxels of the scene. Based on the prediction results output by each head network in the output network and the label information corresponding to the batch of sensor data, the loss value of each head network in the output network is obtained. The loss values ​​of each head network in the output network are weighted to obtain a loss value corresponding to this training process. This loss value is then used for backpropagation to sequentially update the parameters of the output network, feature fusion network, and feature extraction network (including image feature extraction network and point cloud feature extraction network). It will be understood that Figure 7B is only an illustration of the training process of the perception detection network and is not limited to the training process of the perception detection network shown in Figure 7B. For example, each head network in the output network can also be trained separately.

[0235] Here, the training of the perception detection network may be specifically referred to the relevant description of the embodiment of FIG7A above, which will not be repeated here.

[0236] In some possible embodiments, after the training of the perception detection network is completed, the attribute recognition network may be trained. The following describes the training process of the attribute recognition network.

[0237] Taking the attribute recognition network shown in Figure 2 as an example, the attribute recognition network includes a text encoding network and an attribute decoding network. Exemplarily, the text encoding network can directly use a trained word vector feature extractor (or obtain it by pre-training learning of text-image), and the embodiment of the present application can only train the attribute decoding network. The trained word vector feature extractor can extract the word vector features of the text query information based on the input text query information, and the word vector features of the text query information can represent the image semantic features of the category indicated by the text query information.

[0238] Exemplarily, the process of obtaining a text encoding network using text-image pre-training can be: obtaining a large amount of text-image training data, wherein the text-image training data includes multiple text-image data groups, each text-image data group includes text information indicating category information and an image corresponding to the text information, for example, text-image training group 1 includes text information indicating a car and an image of a car, the text information in the text-image training data is input into the text encoder to extract the word vector features of each text information respectively, the image in the text information in the text-image training data is input into the image encoder to extract the image features of each image respectively, based on the training idea of ​​making the word vector features of the text information belonging to the same text-image data group as close as possible to the image features of the image, and the word vector features of the text information belonging to different text-image data groups as far away from the image features of the image as possible, the parameters of the text encoder and the parameters of the image encoder are adjusted, and the trained text encoder can be directly used as the text encoding network in the above-mentioned attribute recognition network.

[0239] For example, the training of the attribute decoding network can be as follows: the text encoding network inputs the word vector features of each text query information to the attribute decoding network based on the received multiple text query information; the attribute decoding network predicts the category information of each obstacle based on the word vector features of the multiple text query information and the fusion features of the voxels of the obstacle provided by the perception detection network; the loss value of the attribute decoding network for this training is obtained based on the predicted category information of each obstacle and the category labeling information of each obstacle; and finally, the parameters of the attribute decoding network are reversely updated based on the loss value of the attribute decoding network for this training.

[0240] In some possible embodiments, after the perception detection network is trained, the path evaluation network may be trained. The following describes the training process of the path evaluation network.

[0241] Taking the path evaluation network shown in Figure 2 as an example, the path evaluation network includes a path encoding network, a feature interaction network, and an evaluation output network. The training process of the path evaluation network may, for example, include: obtaining path training data, which includes multiple paths planned by the vehicle within the aforementioned K time ranges and recommendation coefficient annotation information for these multiple paths; inputting these multiple paths into the path encoding network; the path encoding network outputting the extracted path features of each path into the feature interaction network; the feature interaction network outputting a risk signature for each path based on the path features of each path and the fused voxel features of the scene at the aforementioned K time periods (from the perception detection network); the evaluation output network outputting predicted recommendation coefficients for the multiple paths based on the risk signatures of the multiple paths; and determining a predicted recommended path from the multiple paths based on the predicted recommendation coefficients of the multiple paths; obtaining loss values ​​for the multiple paths based on the predicted recommendation coefficients and the recommendation coefficient annotation information for the multiple paths, wherein the loss values ​​for the multiple paths are obtained by weighting the loss values ​​of each path in the multiple paths; and finally, updating parameters in the path evaluation network based on the loss values ​​of the multiple paths.

[0242] It can be seen that the training of the above-mentioned perception detection network, attribute recognition network, and path evaluation network is carried out separately and independently. In some possible embodiments, the perception detection network, attribute recognition network, and path evaluation network can also be trained jointly. In this case, the loss value corresponding to each training process is obtained by weighting the loss value of the attribute recognition network in that training process, the loss value of the perception detection network in that training process (i.e., the loss value of the attribute decoding network in that training process), and the loss value of the path evaluation network in that training process. Finally, the loss value of that training process can be used to update the parameters of the above-mentioned perception detection network, the attribute decoding network in the attribute recognition network, and the above-mentioned path evaluation network.

[0243] Please refer to Figure 8, which is a schematic diagram of a chip hardware structure provided in an embodiment of the present application, which can be used to execute the intelligent driving method and / or training method in an embodiment of the present application.

[0244] As shown in FIG8 , a neural-networks processing unit (NPU) 80 is mounted on a host CPU (Host CPU) as a coprocessor, and the Host CPU assigns tasks to execute the intelligent driving method in the aforementioned embodiment or the related processes of the training method in the aforementioned embodiment.

[0245] The core part of the NPU is the operation circuit 803, and the controller 804 controls the operation circuit 803 to extract data from the memory (weight memory or input memory) and perform operations.

[0246] In some implementations, the arithmetic circuit 803 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 803 is a two-dimensional systolic array. The arithmetic circuit 803 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 803 is a general-purpose matrix processor.

[0247] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 802 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 801 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 808.

[0248] The vector calculation unit 807 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 807 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0249] In some implementations, the vector calculation unit 807 can store the processed output vector to the unified memory 806. For example, the vector calculation unit 807 can apply a nonlinear function to the output of the operation circuit 803, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 807 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 803, for example, for use in a subsequent layer in a neural network.

[0250] The unified memory 806 is used to store input data and output data.

[0251] The direct memory access controller 805 (DMAC) moves the input data in the external memory to the input memory 801 and / or the unified memory 806, stores the weight data in the external memory into the weight memory 802, and stores the data in the unified memory 806 into the external memory.

[0252] The bus interface unit (BIU) 810 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 809 through the bus.

[0253] An instruction fetch buffer 809 connected to the controller 804 is used to store instructions used by the controller 804 .

[0254] The controller 804 is used to call the instructions cached in the instruction fetch memory 809 to control the working process of the computing accelerator.

[0255] Generally, the unified memory 806, the input memory 801, the weight memory 802 and the instruction fetch memory 809 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0256] 9A , which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application, the computing device 30 includes a receiving unit 310 and a processing unit 312. The computing device 30 can be implemented by hardware, software, or a combination of hardware and software.

[0257] Among them, the receiving unit 310 is used to obtain the data collected by the sensor on the first scene, and the sensor includes at least one of a camera and a radar; the processing unit 312 is used to input the collected data into the perception detection network and output perception information, which is used to indicate the voxels of the obstacles in the first scene; the processing unit 312 is also used to display the voxels of the obstacles based on at least the perception information.

[0258] In some possible embodiments, the computing device 30 further includes a display unit 314 (not shown), which is used to: display the above-mentioned obstacle based on the perception information, where the obstacle is marked with a polygonal box; and / or display the voxels of the above-mentioned obstacle based on the perception information.

[0259] The computing device 30 may be used to implement the method described in the embodiment of Figure 4. In the embodiment of Figure 4, the receiving unit 310 may be used to execute S401, and the processing unit 312 may be used to execute S402 and S403.

[0260] 9B , which is a schematic diagram of the structure of a training device provided in an embodiment of the present application, the training device 40 includes an encoding unit 410, a decoding unit 412, and an updating unit 414. The training device 40 can be implemented by hardware, software, or a combination of hardware and software.

[0261] The encoding unit 410 is used to extract features from the image data at each moment in a batch of sensor data through an image feature extraction network during each training process to obtain 3D image features of the image data at K moments, where K is a positive integer; extract features from the point cloud data at each moment in a batch of sensor data through a point cloud feature extraction network to obtain point cloud features of voxels corresponding to the point cloud data at K moments; and fuse the 3D image features of the image data at K moments and the point cloud features of the voxels corresponding to the point cloud data at K moments through a feature fusion network to obtain field features at K moments. The decoding unit 412 is used to output K prediction results based on the fusion features of the voxels of the scene at K moments through each head network in the output network, wherein each of the K prediction results corresponds to the scene at one moment; the updating unit 414 is used to obtain the loss value of each head network in the output network according to the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network; and to weight the loss values ​​of each head network in the output network to obtain a loss value corresponding to each training process; and to use the loss value to update the parameters in the perception detection network.

[0262] The training device 40 can be used to implement the method described in the embodiment of Figure 7A. In the embodiment of Figure 7A, the encoding unit 410 can be used to perform S701-S703, the decoding unit 412 can be used to perform S704, and the updating unit 414 can be used to perform S705 and S706.

[0263] It should be understood that the division of the various units in the above devices (e.g., computing device 30 and training device 40) is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity, or physically separated. Furthermore, the units in the device may be implemented in the form of a processor invoking software; for example, the device may include a processor connected to a memory storing instructions, and the processor invoking the instructions stored in the memory to implement any of the above methods or the functions of the various units in the device. The processor may be, for example, a general-purpose processor such as a central processing unit (CPU) or a microprocessor, and the memory may be internal or external to the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), which realizes the functions of some or all of the above units by designing the logical relationship of the components in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above units. All units of the above devices can be implemented in the form of software called by the processor, or in the form of hardware circuits, or in part by software called by the processor, and the rest by hardware circuits.

[0264] In an embodiment of the present application, a processor is a circuit having a signal processing capability. In one implementation, the processor may be a circuit having the capability to read and execute instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by a processor as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration process, which may be understood as the processor loading instructions to implement the functions of some or all of the above units. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0265] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0266] In addition, the various units in the above devices can be fully or partially integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the device. The type of the at least one processor can be different, for example, including a CPU and FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0267] Referring to Figure 10 , Figure 10 is a schematic diagram of the structure of a processing device provided in an embodiment of the present application. As shown in Figure 10 , processing device 50 includes: a processor 501, a communication interface 502, a memory 503, and a bus 504. Processor 501, memory 503, and communication interface 502 communicate with each other via bus 504. It should be understood that this application does not limit the number of processors and memories in processing device 50.

[0268] In one implementation, the processing device 50 is a component (e.g., a chip or integrated circuit) within a vehicle for autonomous driving control. The vehicle is equipped with an autonomous driving system. The autonomous driving system is not limited to fully autonomous driving systems, highly autonomous driving systems, conditionally autonomous driving systems, or partially autonomous driving systems. Those skilled in the art will appreciate that any non-fully manual driving system that provides intelligent driving is encompassed by this concept.

[0269] In another implementation, the processing device 50 may be a network-side device. A network-side device is a device with computing capabilities. The network-side device may be, for example, a server deployed on the network side (e.g., a server for intelligent driving processing), or a component or chip in the server. In some possible embodiments, the network-side device may also be a system-level device composed of multiple servers. The network-side device may be deployed in a cloud environment or an edge environment, which is not specifically limited here.

[0270] Bus 504 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses can be categorized as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 504 may include a path for transmitting information between various components of processing device 50 (e.g., memory 503, processor 501, and communication interface 502).

[0271] The processor 501 can refer to the relevant description of the processor in the above embodiment, which will not be repeated here.

[0272] Memory 503 is used to provide storage space for storing data such as the operating system and computer programs. Memory 503 can be one or a combination of random access memory (RAM), erasable programmable read-only memory (EPROM), read-only memory (ROM), or compact disc read-only memory (CD-ROM). Memory 503 can exist independently or be integrated into processor 501.

[0273] The communication interface 502 can be used to provide information input or output for the processor 501. Alternatively, the communication interface 502 can be used to receive data transmitted externally and / or transmit data externally. It can be a wired link interface such as an Ethernet cable, or a wireless link interface (such as Wi-Fi, Bluetooth, general wireless transmission, etc.). Alternatively, the communication interface 502 can also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0274] In some possible embodiments, the processing device 50 further includes a display 505. The display 505 is connected or coupled to the processor 501 via a bus 504. The display 505 can be used to display polygonal instances of the first scene. The display 505 can be a display screen, and the display screen can be a liquid crystal display (LCD), an organic or inorganic light-emitting diode (OLED), an active matrix organic light-emitting diode panel (AMOLED), etc. The display 505 can also be a car tablet, an in-vehicle display, a head-up display (HUD) system, or an enhanced head-up display AR-HUD system, etc.

[0275] The processor 501 in the processing device 50 is used to read the computer program stored in the memory 503 to execute the aforementioned method, such as the method described in FIG. 4 or FIG. 7A .

[0276] In one possible design, the processing device 50 may be one or more modules in an execution body for executing the method shown in FIG. 4 . The processor 501 may be configured to read one or more computer programs stored in a memory and perform the following operations:

[0277] Acquiring data collected by a sensor on a first scene through a receiving unit 310, where the sensor includes at least one of a camera and a radar;

[0278] The collected data is input into a perception detection network, and perception information is output, where the perception information is used to indicate voxels of obstacles in the first scene; and the voxels of the obstacles are displayed at least based on the perception information.

[0279] In another possible design, the processing device 50 may be one or more modules in an execution body that executes the method shown in FIG. 7A . The processor 501 may be configured to read one or more computer programs stored in a memory to perform the following operations:

[0280] During each training process, the encoding unit 410 performs feature extraction on the image data at each moment in a batch of sensor data through an image feature extraction network to obtain 3D image features of the image data at K moments, where K is a positive integer. During each training process, the point cloud feature extraction network performs feature extraction on the point cloud data at each moment in a batch of sensor data to obtain point cloud features of voxels corresponding to the point cloud data at the K moments. Furthermore, the feature fusion network performs feature fusion on the 3D image features of the image data at the K moments and the point cloud features of the voxels corresponding to the point cloud data at the K moments to obtain fused voxel features of the scene at the K moments.

[0281] Outputting K prediction results through each head network in the output network according to the fusion features of the voxels of the scenes at K moments through the decoding unit 412, wherein each prediction result of the K prediction results corresponds to a scene at one moment;

[0282] The update unit 414 obtains the loss value of each head network in the output network based on the label information corresponding to the batch of sensor data and the K prediction results output by each head network in the output network; and the loss values ​​of each head network in the output network are weighted to obtain a loss value corresponding to each training process; and the loss value is used to update the parameters in the perception detection network.

[0283] In the embodiments described above, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a particular embodiment, please refer to the relevant descriptions of other embodiments. In addition, in the various embodiments of this application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0284] It should be noted that, those skilled in the art can see that all or part of the steps in the various methods of the above embodiments can be completed by a program to instruct relevant hardware. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0285] The technical solution of the present application may essentially or contribute to the part or all or part of the technical solution in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a device (which may be a personal computer, a server, or a network device, a robot, a single-chip microcomputer, a chip, a robot, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

Claims

1. An intelligent driving method, characterized in that: The method comprises: Acquire data collected by a sensor on a first scene, wherein the sensor includes at least one of a camera and a radar; Inputting the collected data into a perception detection network and outputting perception information, wherein the perception information is used to indicate voxels of obstacles in the first scene; The vehicle driving is controlled based at least on the perception information.

2. The method according to claim 1, characterized in that The method further comprises: Displaying the obstacle based on the perception information, wherein the obstacle is marked with a polygonal frame; and / or Voxels of the obstacle are displayed based on the sensory information.

3. The method according to claim 1 or 2, characterized in that: The perception information includes at least one of the following information: The occupation state of the voxels of the first scene, the velocity information of the voxels of the first scene, the visibility state of the voxels of the first scene and the corner point information of the polygonal box corresponding to the obstacle; The polygonal frame corresponding to the obstacle is associated with the voxel of the obstacle.

4. The method according to any one of claims 1 to 3, characterized in that: The perception information is also used to indicate voxels of a road surface of the first scene, and the controlling the vehicle driving based at least on the perception information includes: generating road surface geometry information of the first scene at least based on the perception information; The suspension in the vehicle is adjusted based on the road geometry information.

5. The method according to any one of claims 1 to 4, characterized in that: The controlling the vehicle driving at least based on the perception information includes: adjusting the driving path of the vehicle at least according to the perception information, wherein the adjusted driving path does not pass through the area where the voxels of the obstacle are located.

6. The method according to any one of claims 1 to 5, characterized in that: The collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network and an output network, wherein: The image feature extraction network is used to extract 3D image features of the image data; The point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data; The feature fusion network is used to fuse the point cloud features of the voxels corresponding to the 3D image features and the point cloud data to obtain fused features of the voxels of the first scene; The output network is used to process the fused features of the voxels of the first scene and output the perception information.

7. The method according to claim 6, characterized in that The method further comprises: Inputting the text query information and the fusion features of the voxels of the obstacle into the attribute recognition network, and outputting the category information of the obstacle; the text query information is used to request the query category; Displaying category information of the obstacle; The fusion features of the voxels of the obstacle are determined based on the corner point information of the polygonal box corresponding to the obstacle and the fusion features of the voxels of the first scene, and the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle.

8. The method according to claim 6 or 7, characterized in that: The method further comprises: Obtain multiple planned paths for the vehicle; Inputting the fused features of the multiple planned paths of the vehicle and the voxels of the first scene into a path evaluation network, outputting recommendation coefficients of the multiple planned paths and a recommended path among the multiple planned paths, wherein the recommended path is associated with the recommendation coefficients of the multiple planned paths; The recommended route is displayed.

9. A system for intelligent driving, characterized in that: The system comprises: A perception detection network, configured to output perception information based on data collected by a sensor on a first scene, wherein the perception information is used to indicate voxels of obstacles in the first scene; the sensor includes at least one of a camera and a radar; an attribute recognition network, configured to output category information of the obstacle according to text query information and fusion features of voxels of the obstacle, wherein the fusion features of the voxels of the obstacle are determined based on corner point information of a polygonal box corresponding to the obstacle and fusion features of voxels of the first scene, wherein the polygonal box corresponding to the obstacle is associated with the voxels of the obstacle, and the fusion features of the voxels of the first scene are obtained by the perception detection network through temporal and / or spatial fusion based on at least one of 3D image features extracted from the acquired data and point cloud features of voxels; A path evaluation network is used to output recommendation coefficients of the multiple planned paths and recommended paths among the multiple planned paths based on fusion features of the multiple planned paths and voxels of the first scene, wherein the recommended paths are associated with the recommendation coefficients of the multiple planned paths.

10. The system according to claim 9, characterized in that The perception information includes at least one of the following information: The occupation state of the voxels of the first scene, the velocity information of the voxels of the first scene, the visibility state of the voxels of the first scene and the corner point information of the polygonal box corresponding to the obstacle; The polygonal frame corresponding to the obstacle is associated with the voxel of the obstacle.

11. The system according to claim 9 or 10, characterized in that: The collected data includes image data and point cloud data, and the perception detection network includes an image feature extraction network, a point cloud feature extraction network, a feature fusion network and an output network, wherein: The image feature extraction network is used to extract 3D image features of the image data; The point cloud feature extraction network is used to extract point cloud features of voxels corresponding to the point cloud data; The feature fusion network is used to fuse the point cloud features of the voxels corresponding to the 3D image features and the point cloud data to obtain fused features of the voxels of the first scene; The output network is used to process the fused features of the voxels of the first scene and output the perception information.

12. The system according to any one of claims 9 to 11, characterized in that: The attribute recognition network includes a text encoding network and an attribute decoding network, wherein: The text encoding network is used to extract word vector features of text query information; The attribute decoding network is used to output the category information of the obstacle according to the fusion features of the word vector features and the voxels of the obstacle.

13. The system according to any one of claims 9 to 12, characterized in that: The path evaluation network includes a path encoding network, a feature interaction network and an evaluation output network, wherein: The path encoding network is used to extract path features of each planned path among the multiple planned paths; The feature interaction network is used to obtain the risk feature of each planned path according to the path feature of each planned path and the fusion feature of the voxels of the first scene; The evaluation output network is used to output the recommendation coefficients of the multiple planned paths and the recommended paths among the multiple planned paths according to the risk characteristics of the multiple planned paths.

14. A device for intelligent driving, characterized in that: The device comprises: A receiving unit, configured to obtain data collected by a sensor on a first scene, wherein the sensor includes at least one of a camera and a radar; a processing unit, configured to input the collected data into a perception detection network and output perception information, wherein the perception information is used to indicate voxels of obstacles in the first scene; The processing unit is further used to control vehicle driving based at least on the perception information.

15. A computing device, characterized in that: The computing device comprises a memory and a processor, wherein the memory is used to store program instructions; when the processor executes the program instructions in the memory, the computing device executes the method according to any one of claims 1 to 8.

16. A vehicle, characterized in that: The vehicle comprises the system as described in any one of claims 9 to 13, or comprises the device as described in claim 14 or 15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions are used to implement the method according to any one of claims 1 to 8.

Images