Autonomous driving method and system, and medium, infrastructure-side server and smart device

Abstract

The present application relates to the technical field of autonomous driving. Specifically provided are an autonomous driving method and system, and a medium, an infrastructure-side server and a smart device, which aim to solve the problem of how to achieve robust autonomous driving capabilities while reducing the cost per vehicle and avoiding the waste of vehicle-side resources. To this end, in the present application, multimodal perception fusion is performed on the basis of data collected by various types of sensors that are arranged on an infrastructure side, so as to obtain an infrastructure-side target perception result; environmental fusion is performed on the infrastructure-side target perception result and an environmental map to obtain an infrastructure-side environmental fusion result; and a trajectory planning result of a smart device is obtained on the basis of the infrastructure-side environment fusion result. A smart device realizes driving control on the basis of a trajectory planning result and a safety guard decision result, such that lightweight deployment on a smart-device side is realized, and safety protection of the smart-device side is also ensured. The present application is applicable to various different types of smart devices and various smart devices having low computing power, which makes obtained trajectory planning results more robust.

Description

Autonomous driving method, system, medium, field server and intelligent device

[0001] This application claims priority to Chinese patent application CN 202410347102.4 filed on March 26, 2024, with the invention name “Autonomous driving method, system, medium, field server and intelligent device”. The entire contents of the above Chinese patent application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of autonomous driving technology, and specifically provides an autonomous driving method, system, medium, field server and intelligent device. Background Art

[0003] Scenario-specific autonomous driving solutions are gaining increasing attention. Existing technologies that rely solely on vehicle deployment often increase vehicle costs and leave computing resources and sensors idle and wasted.

[0004] Accordingly, this field requires a new autonomous driving solution to solve the above problems.

[0005] Summary of the Invention

[0006] In order to overcome the above-mentioned defects, the present application is proposed to provide a solution or at least partially solve the problem of how to achieve robust autonomous driving capabilities while reducing the cost of each vehicle and avoiding waste of vehicle-side resources.

[0007] In a first aspect, the present application provides an autonomous driving method, which is applied to a field terminal and includes:

[0008] Based on data collected by at least two types of sensors provided at the field end, multimodal perception fusion is performed to obtain a field end target perception result; wherein the field end target perception result includes an intelligent device for which trajectory planning is to be performed; and the intelligent device is communicatively connected to the field end;

[0009] Performing environmental fusion based on the field-side target perception result and the environmental map to obtain a field-side environmental fusion result;

[0010] Performing trajectory planning on the smart device according to the field-side environment fusion result to obtain a trajectory planning result of the smart device;

[0011] The trajectory planning result is sent to the smart device, so that the smart device can realize the automatic driving function according to the trajectory planning result.

[0012] In one technical solution of the above-mentioned autonomous driving method, performing trajectory planning for the smart device based on the field-side environment fusion result to obtain the trajectory planning result of the smart device includes:

[0013] Obtaining status data and scheduling data of the smart device;

[0014] According to the state data, the scheduling data and the field environment fusion result, the trajectory of the smart device is planned to obtain the trajectory planning result.

[0015] In one technical solution of the above-mentioned autonomous driving method, obtaining the trajectory planning result includes:

[0016] Obtain the planned trajectory and real-time pose of the smart device.

[0017] In one technical solution of the above-mentioned autonomous driving method, the method further includes:

[0018] Achieving communication connection between the field terminal and the smart device via a wireless link;

[0019] The wireless link has time synchronization capability.

[0020] In one technical solution of the above-mentioned autonomous driving method, the wireless link is a redundant wireless link; and the method further includes:

[0021] The redundant wireless links are implemented according to a variety of different types of wireless communication protocols.

[0022] In one technical solution of the above-mentioned autonomous driving method, the multimodal fusion is implemented by a multimodal fusion module; the environmental fusion is implemented by an environmental fusion module; and the trajectory planning is implemented by a trajectory planning module. The method further includes:

[0023] Performing fault monitoring on the sensor, the multimodal fusion module, the environment fusion module, and the trajectory planning module;

[0024] When at least one of the sensor, the multimodal fusion module, the environment fusion module and the trajectory planning module fails, a fault self-correction mechanism responds and fault information corresponding to the fault is saved.

[0025] In one technical solution of the above-mentioned autonomous driving method, the sensor includes a first sensor and a second sensor, and the number of the first sensor and the number of the second sensor are both plural;

[0026] The performing multimodal perception fusion based on data collected by at least two types of sensors provided at the field end to obtain a field end target perception result includes:

[0027] Acquire a first perception feature based on data collected by the first sensor;

[0028] acquiring a second perception feature based on data collected by the second sensor;

[0029] Performing feature fusion based on the first perception feature and the second perception feature to obtain a perception fusion feature;

[0030] The field-side target perception result is obtained according to the perception fusion feature.

[0031] In one technical solution of the above-mentioned autonomous driving method, obtaining the field-side target perception result includes:

[0032] According to the perception fusion feature, the dynamic target perception result of the environment, the static obstacle occupancy result and the positioning result of the smart device are obtained.

[0033] In one technical solution of the above-mentioned autonomous driving method, the method further includes obtaining the environment map according to the following steps:

[0034] Based on the data collected by the sensor, acquiring the field environment data of the field end;

[0035] Based on the field environment data, acquiring static obstacle occupancy data;

[0036] The environment map is obtained according to the static obstacle occupancy data.

[0037] In one technical solution of the above-mentioned autonomous driving method, the method further includes:

[0038] updating the environment map according to the return data;

[0039] The reflow data is data collected by sensors of the field end or the smart device that triggers reflow based on preset rules.

[0040] In one technical solution of the above-mentioned autonomous driving method, the method further includes:

[0041] Performing on-site calibration or online calibration on the sensor to obtain an external parameter calibration result of the sensor;

[0042] The external parameter calibration result is used for the multimodal perception fusion and / or the environment fusion.

[0043] In a second aspect, the present application provides an autonomous driving method, which is applied to a smart device and includes:

[0044] Obtaining a trajectory planning result of the smart device sent by a field end; wherein the field end is in communication with the smart device;

[0045] Controlling the driving of the smart device according to the trajectory planning result and the safety guard decision result of the smart device to realize the automatic driving function of the smart device;

[0046] The safety guard decision result is a result obtained after the smart device performs safety control evaluation and verification on the trajectory planning result.

[0047] In one technical solution of the above-mentioned autonomous driving method, controlling the driving of the smart device based on the trajectory planning result and the safety guard decision result of the smart device includes:

[0048] Performing priority judgment based on the trajectory planning result and the security guard decision result of the smart device to obtain a priority judgment result;

[0049] Driving control of the smart device is performed based on the priority judgment result and the odometer fusion positioning data of the smart device.

[0050] In one technical solution of the above-mentioned autonomous driving method, controlling the driving of the smart device based on the priority determination result and the odometer-fused positioning data of the smart device includes:

[0051] When the priority of the trajectory planning result is higher than the priority of the safety guard decision result, controlling the driving of the smart device according to the trajectory planning result and the odometer fused positioning data;

[0052] When the priority of the safety guard decision result is higher than the priority of the trajectory planning result, the driving of the smart device is controlled according to the safety guard decision result and the odometer fusion positioning data.

[0053] In one technical solution of the above-mentioned autonomous driving method, the smart device is communicatively connected to the field terminal via a wireless link, and the wireless link has a time synchronization capability;

[0054] The step of controlling the driving of the smart device according to the trajectory planning result and the odometer-fused positioning data includes:

[0055] Time synchronization of the trajectory planning result via the wireless link;

[0056] Based on the time-synchronized trajectory planning result and the odometer fusion positioning data, the driving of the smart device is controlled.

[0057] In one technical solution of the above-mentioned autonomous driving method, the trajectory planning result includes the planned trajectory and real-time pose of the smart device;

[0058] The method of controlling the driving of the smart device based on the time-synchronized trajectory planning result and the odometer fusion positioning data includes:

[0059] Perform motion compensation according to the real-time posture after time synchronization of the odometer fusion positioning data to obtain a compensated real-time posture;

[0060] The intelligent device is controlled to travel based on the compensated real-time posture and the time-synchronized planned trajectory.

[0061] In one technical solution of the above-mentioned autonomous driving method, the wireless link is a redundant wireless link; and the method further includes:

[0062] The redundant wireless links are implemented according to a variety of different types of wireless communication protocols.

[0063] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the autonomous driving method described in any one of the technical solutions of the above-mentioned autonomous driving method.

[0064] In a fourth aspect, a field-side server is provided, the field-side server comprising:

[0065] at least one processor;

[0066] and, a memory communicatively coupled to the at least one processor;

[0067] In which, the memory stores a computer program, and when the computer program is executed by the at least one processor, it implements the autonomous driving method described in any one of the technical solutions of the above-mentioned autonomous driving method.

[0068] In a fifth aspect, a smart device is provided, comprising:

[0069] at least one processor;

[0070] and, a memory communicatively coupled to the at least one processor;

[0071] In which, the memory stores a computer program, and when the computer program is executed by the at least one processor, it implements the autonomous driving method described in any one of the technical solutions of the above-mentioned autonomous driving method.

[0072] In a sixth aspect, an autonomous driving system is provided, comprising the field-side server in the above-mentioned field-side server technical solution and the smart device in the smart device technical solution, wherein the field-side server is communicatively connected to the smart device.

[0073] The above one or more technical solutions of this application have at least one or more of the following beneficial effects:

[0074] In the technical solution for implementing this application, this application performs multimodal perception fusion based on data collected by various types of sensors set up on the field side, obtains field-side target perception results, performs environmental fusion based on the field-side target perception results and the environment map, obtains field-side environment fusion results, implements trajectory planning for smart devices based on the field-side environment fusion results, obtains trajectory planning results for smart devices, and sends them to smart devices, thereby enabling smart devices to implement autonomous driving functions based on trajectory planning results. Through the above configuration method, this application can realize the perception, fusion, planning and other functions of autonomous driving through the field side, without the need for smart devices to deploy complex perception, fusion, rules and other functions, to realize the autonomous driving function of smart devices, avoids the idleness and waste of some computing resources and sensors on the smart device side when the autonomous driving function is not in use, can reduce the cost of a single vehicle, and is applicable to many different types of smart devices and various low-computing-power smart devices. At the same time, because the field side realizes multimodal perception fusion based on data collected by multiple types of sensors, it combines the advantages of multiple sensors and can adapt to a variety of different weather and environmental conditions, making the trajectory planning results more robust. At the same time, smart devices achieve driving control based on the trajectory planning results and safety guard decision results issued by the field side. This can achieve lightweight deployment on the smart device side while ensuring safety protection on the smart device side. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Among them:

[0076] FIG1 is a schematic flow chart of main steps of an autonomous driving method according to an embodiment of the present application;

[0077] FIG2 is a schematic flow chart of main steps of an autonomous driving method according to another embodiment of the present application;

[0078] FIG3 is a schematic diagram of the main hardware topology structure between the field end and the smart device according to an implementation of an embodiment of the present application;

[0079] FIG4 is a schematic diagram of the main software architecture of a battery swap station according to an implementation method of an embodiment of the present application;

[0080] FIG5 is a schematic diagram of the main software architecture of the vehicle side according to an implementation method of an embodiment of the present application;

[0081] FIG6 is a schematic diagram of the main framework for obtaining an environment map according to an implementation of an embodiment of the present application. DETAILED DESCRIPTION

[0082] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0083] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0084] Refer to Figure 1, which is a schematic flow chart of the main steps of an autonomous driving method according to an embodiment of the present application. As shown in Figure 1, the autonomous driving method in this embodiment of the present application is applied to the field end and mainly includes the following steps S101-S104.

[0085] Step S101: Based on data collected by at least two types of sensors set up at the field end, multimodal perception fusion is performed to obtain a field end target perception result; wherein the field end target perception result includes the smart device to be trajectory planned; the smart device is communicatively connected with the field end.

[0086] In this embodiment, the field side can be equipped with multiple different types of sensors for collecting environmental data. Multimodal perception fusion can be performed based on the data collected by these different types of sensors to obtain field side target perception results. These field side target perception results include the smart device for which trajectory planning is to be performed, and the smart device is in communication with the field side.

[0087] In one embodiment, the field end may be a space where sensors can be installed and deployed, such as a battery swap station, a charging station, or a smart parking lot.

[0088] In one embodiment, the smart device may be a driving device, a smart car, a robot, or the like.

[0089] In one embodiment, the smart device to be trajectory planned may be a vehicle to be replaced with a battery, a vehicle to be parked, etc.

[0090] In one embodiment, the sensor may include a lidar, a camera, a millimeter-wave radar, etc.

[0091] In one implementation, multiple sensors can be deployed based on the functional requirements and coverage of the field, taking into account the performance of different sensors. Multiple deployments of each type are possible. Multimodal perception fusion based on multiple sensor types can improve the robustness of environmental perception in complex weather conditions, enhance the semantic recognition of perceived targets within the field environment, and reduce perception performance risks such as false detection and missed detection.

[0092] In one embodiment, the field-side target perception results may include environmental dynamic target perception results, static obstacle occupancy results, and smart device positioning results.

[0093] Step S102: Perform environmental fusion based on the field-side target perception result and the environment map to obtain a field-side environmental fusion result.

[0094] In this embodiment, environment fusion can be performed based on the field-side target perception result and the environment map, thereby obtaining a field-side environment fusion result.

[0095] In one embodiment, the field-side environment fusion result may include the result of fusing the environment dynamic target perception result and the positioning result of the smart device with the environment.

[0096] Step S103: According to the field environment fusion result, the trajectory of the smart device is planned to obtain the trajectory planning result of the smart device.

[0097] In this embodiment, the trajectory of the smart device can be planned based on the field environment fusion result, thereby obtaining the trajectory planning result of the smart device.

[0098] In one embodiment, the trajectory planning result may include the planned trajectory and real-time pose (Ego Motion) of the smart device.

[0099] Step S104: Send the trajectory planning result to the smart device, so that the smart device can realize the automatic driving function according to the trajectory planning result.

[0100] In this embodiment, the field end can send the obtained trajectory planning results to the smart device that is communicatively connected to the field end, so that the smart device can realize the automatic driving function according to the trajectory planning results.

[0101] Based on the above steps S101 to S104, the embodiment of the present application performs multimodal perception fusion based on the data collected by various types of sensors set up on the field side, obtains the field side target perception result, performs environmental fusion based on the field side target perception result and the environment map, obtains the field side environment fusion result, implements trajectory planning for the smart device based on the field side environment fusion result, obtains the trajectory planning result of the smart device, and sends it to the smart device, thereby enabling the smart device to implement the autonomous driving function according to the trajectory planning result. Through the above configuration, the embodiment of the present application can realize the perception, fusion, planning and other functions of autonomous driving through the field side, without the deployment of complex perception, fusion, rules and other functions on the smart device, and can realize the autonomous driving function of the smart device, avoiding the idleness and waste of some computing resources and sensors on the smart device side when the autonomous driving function is not in use, which can reduce the cost of each vehicle and is applicable to various types of smart devices and various low-computing-power smart devices. At the same time, because the field side realizes multimodal perception fusion based on data collected by multiple types of sensors, it combines the advantages of multiple sensors and can adapt to a variety of different weather and environmental conditions, so that the trajectory planning result obtained has higher robustness.

[0102] Step S101, step S102 and step S103 are further described below.

[0103] In one embodiment of the application, the sensor may include a first sensor and a second sensor; step S101 may further include the following steps S1011 to S1014:

[0104] Step S1011: Acquire a first perception feature based on data collected by the first sensor.

[0105] Step S1012: Acquire a second perception feature based on data collected by the second sensor.

[0106] Step S1013: Perform feature fusion based on the first perception feature and the second perception feature to obtain a perception fusion feature.

[0107] Step S1014: Obtaining the field-side target perception result based on the perception fusion feature.

[0108] In this embodiment, a first perception feature can be obtained based on data collected by a first sensor, and a second perception feature can be obtained based on data collected by a second sensor to obtain a second bird's-eye view feature. The first and second perception features are fused to obtain a fusion perception feature, and a field-side target perception result is obtained based on the fusion perception feature. The first and second sensors can be cameras, lidars, millimeter-wave radars, and other sensors, as long as they are of different types.

[0109] In one embodiment, feature fusion between the first perception feature and the second perception feature may be achieved based on a neural network model.

[0110] In one embodiment, the neural network model can be a model built based on the Transformer BEV architecture, a model built based on mono 3D, a model built based on the OCC network, etc.

[0111] In one embodiment, the field-side target perception result can be obtained based on the perception fusion characteristics and the state information of the smart device. The state information may include physical state data such as shape parameters and speed.

[0112] In one implementation of the embodiment of the present application, an environment map may be obtained based on steps S201 to S203:

[0113] Step S201: Based on the data collected by the sensor, the field environment data of the field is obtained.

[0114] Step S202: Based on the field environment data, obtain static obstacle occupancy data.

[0115] Step S203: Obtain an environment map based on the static obstacle occupancy data.

[0116] In one embodiment, the environment map may be updated according to the following steps S204:

[0117] Step S204: updating the environment map according to the backflow data; wherein the backflow data is data collected by sensors of the field end or smart device that triggers the backflow based on preset rules.

[0118] The following describes the process of acquiring and updating the environment map with reference to Figure 6, which is a schematic diagram of the main framework for acquiring the environment map according to one embodiment of the present application. As shown in Figure 6, sensors installed at the site include cameras and lidars. For example, N cameras and N lidars may be installed. Site-side environmental data is obtained based on the raw image data collected by the cameras and the raw point cloud data collected by the lidars. Based on the site-side environmental data, environmental perception is performed using a perception algorithm to obtain static obstacle occupancy data. The perception algorithm may be a neural network-based perception algorithm. Specifically, the perception algorithm performs lidar target detection based on the raw point cloud data to obtain voxelized features, performs camera target detection based on the raw image data to obtain camera features, and then fuses the voxelized features and camera features using a neural network to obtain static obstacle occupancy data. The static obstacle occupancy data may include occupancy data for immovable static obstacles such as curbs, walls, and tree stumps. Based on the static obstacle occupancy data, an environment map can be obtained. The environment map represents the boundaries of the static obstacles. Construction of the environment map can be implemented in the cloud. The large model of the perception cloud outputs the corresponding static obstacle occupancy data; the cloud management platform of the environmental map on the cloud realizes environmental map inspection, static obstacle occupancy data annotation in the environmental map, and triggers the issuance of the environmental map. The field end is connected to the cloud through a 4G router. The environmental fusion module on the field end obtains the environmental map from the cloud or updates and replaces the environmental map, and then implements environmental modeling and environmental fusion based on the environmental map. At the same time, the field end is also provided with a data mining module, which is used to trigger the return data based on preset rules. The data collected by the sensors of the return field end or smart devices is transmitted to the cloud for updating the environmental map. The environmental map construction and update mechanism combining the cloud and the field end can improve the robustness of environmental fusion.

[0119] In one implementation of the embodiment of the present application, step S103 may further include the following steps S1031 and S1032:

[0120] Step S1031: Acquire status data and scheduling data of the smart device.

[0121] Step S1032: Based on the fusion results of the status data, scheduling data and field environment, trajectory planning is performed on the smart device to obtain a trajectory planning result.

[0122] In this embodiment, the state data and scheduling data of the smart device can be obtained. Based on the state data, scheduling data, and the field environment fusion results, the trajectory of the smart device is planned to obtain a trajectory planning result. The state data of the smart device can include the functional state machine data of the smart device and physical state data such as shape parameters and speed.

[0123] Among them, the trajectory planning can be performed using a planning and control algorithm (PNC, Planning and Controlling) commonly used in this field, and this application does not limit this.

[0124] In one embodiment of the present application, a wireless link can be used to achieve communication between the field end and the smart device. The wireless link has a time synchronization capability, which can facilitate time synchronization between the field end and the smart device via the wireless link, reduce the impact of the wireless link on the transmission process and subsequent control processes, and improve the consistency of performance between the field end and the smart device.

[0125] In one embodiment, the wireless links are redundant, i.e., multiple wireless links are implemented using a variety of different wireless communication protocols. These protocols may include WiFi, 4G / 5G, V2X, and others. The provision of redundant wireless links can further enhance the robustness and real-time performance of communication between the field and smart devices.

[0126] In one implementation of the embodiment of the present application, multimodal fusion can be implemented by a multimodal fusion module; environmental fusion can be implemented by an environmental fusion module; and trajectory planning can be implemented by a trajectory planning module. The field side can also implement fault monitoring of the sensors, multimodal fusion modules, environmental fusion modules, and trajectory planning models set up on the field side. When at least one of the sensors, multimodal fusion modules, environmental fusion modules, and trajectory planning models fails, a fault self-correction mechanism (failsafe) responds, which can support the stability monitoring and operation and maintenance of the field side, and save the fault information corresponding to the fault. For example, the fault information can be uploaded to the cloud for unified management, and can be closed-loop with other service capabilities of the field side (such as battery replacement operation and maintenance, etc.) to improve user experience. Among them, the fault self-correction mechanism refers to a mechanism for complete and full coverage of health monitoring, fault diagnosis, and fault handling at the field side, which can include timely detection of faults and alarm processing, analysis and diagnosis of the causes of faults, targeted fault handling based on the analysis and diagnosis results, and uploading data corresponding to the fault information to the cloud.

[0127] In one embodiment of the present application, the field end also has sensor calibration capabilities. That is, sensors installed at the field end can be calibrated on-site or online to obtain external parameter calibration results for the sensors, which can then be used for multimodal perception fusion and environmental fusion. The field end's sensor calibration capability can prevent the sensor from being affected by factors such as bracket thermal deformation and strong winds, which may cause changes in sensor external parameters. Effective corrections and out-of-tolerance protection can be provided for these situations, thereby ensuring the robustness of multimodal perception fusion and environmental fusion.

[0128] In one embodiment, the field side can also perform registration calculations between the offline point cloud map and the point cloud data collected in real time to correct the external parameter change error between the time when the point cloud data was collected and the current time, thereby obtaining more accurate field side target perception results.

[0129] In one implementation, cloud management equipment can be used to implement OTA upgrades of field-side software, fault operation and maintenance management, upgrades and management of offline environment maps, key data feedback, cloud-based model training, etc.

[0130] Furthermore, the present application provides another autonomous driving method.

[0131] Refer to Figure 2, which is a flow chart of the main steps of an autonomous driving method according to another embodiment of the present application. As shown in Figure 2, in this embodiment, the autonomous driving method is applied to a smart device, and the autonomous driving method may include the following steps S301 to S302:

[0132] Step S301: Obtain the trajectory planning result of the smart device sent by the field end; wherein the field end is in communication connection with the smart device.

[0133] In this embodiment, the smart device can directly obtain the trajectory planning results sent by the field end.

[0134] Step S302: Based on the trajectory planning results and the safety guard decision results of the smart device, the smart device is controlled to realize the automatic driving function of the smart device; wherein the safety guard decision result is the result obtained after the smart device side evaluates and verifies the safety control of the trajectory planning results.

[0135] In this embodiment, the smart device can deploy a safety guard algorithm to implement safety protection processing on the smart device, thereby performing driving control based on the trajectory planning results issued by the field end and the safety guard decision results obtained based on the safety guard algorithm. The safety guard algorithm is used to evaluate and verify the trajectory planning results for safety control. If the trajectory planning results do not meet the safety control instructions of the smart device, the trajectory planning results will be blocked and the smart device will perform safety protection processing. If the trajectory planning results meet the safety control instructions of the smart device, driving control will be performed based on the trajectory planning results. Safety protection processing can include emergency braking, emergency obstacle avoidance, etc.

[0136] In one embodiment, the security guard algorithm may be an RSS (Responsibility Sensitive Safety) algorithm or an SFF (Safety Force Field) algorithm, and the security guard decision result of the smart device may be obtained based on the RSS algorithm or the SFF algorithm.

[0137] Based on the above steps S301-S302, the smart device in the embodiment of the present application needs to deploy a security guard module that can obtain the security guard decision results and obtain the trajectory planning results issued by the field end, so as to realize the automatic driving function, which can effectively realize the lightweight deployment of resources and algorithms on the smart device side, while also ensuring the safety protection of the smart device side.

[0138] Step S302 is further described below.

[0139] In one implementation of the embodiment of the present application, step S302 may further include the following steps S3021 and S3022:

[0140] Step S3021: Perform priority judgment based on the trajectory planning result and the security guard decision result of the smart device to obtain a priority judgment result.

[0141] In this embodiment, the trajectory planning result and the safety guard decision result can be prioritized.

[0142] Step S3022: The smart device is controlled to travel according to the priority judgment result and the odometer-fused positioning data of the smart device.

[0143] In this embodiment, step S3022 may further include the following steps S30221 and S30222:

[0144] Step S30221: When the priority of the trajectory planning result is higher than the priority of the safety guard decision result, the driving of the smart device is controlled according to the trajectory planning result and the odometer fusion positioning data.

[0145] In this embodiment, the smart device and the field end are connected through a wireless link communication, and the wireless link has time synchronization capabilities. When the priority of the trajectory planning result is higher than the priority of the security guard decision result, it means that the trajectory planning result sent by the field end has passed the evaluation and verification of the security control of the smart device end, and the driving control of the smart device can be performed based on the trajectory planning result. The trajectory planning result can be time synchronized based on the wireless link to achieve consistent performance between the field end and the smart device end, and then driving control is performed based on the time-synchronized trajectory planning result and the odometer fusion positioning data. Among them, the odometer fusion positioning data is obtained by fusing the positioning result of the smart device observed by the field end with the local odometry obtained by wheel speed and IMU.

[0146] In one embodiment, the trajectory planning result may include the planned trajectory and real-time pose of the smart device. Motion compensation can be performed on the time-synchronized real-time pose based on fused odometry positioning data to obtain a compensated planned trajectory. Driving control of the smart device is then performed based on the compensated real-time pose and the time-synchronized planned trajectory. Time synchronization and motion compensation enable more precise driving control of the smart device, thereby achieving high-performance autonomous driving capabilities.

[0147] Step S30222: When the priority of the safety guard decision result is higher than the priority of the trajectory planning result, the driving of the smart device is controlled according to the safety guard decision result and the odometer fusion positioning data.

[0148] In this embodiment, when the priority of the safety guard decision result is higher than the priority of the trajectory planning result, it means that the trajectory planning result has not passed the safety control evaluation and verification of the smart device, and the driving control of the smart device cannot be performed based on the trajectory planning result. There may be situations such as the trajectory planning result being invalid. At this time, it is necessary to control the driving of the smart device based on the safety guard decision result and the odometer fusion positioning data, such as emergency braking of the smart device.

[0149] The following, in conjunction with Figures 3 to 5, illustrates the autonomous driving method of the present application, taking the field side as a battery swap station and the smart device as a vehicle as an example. Figure 3 is a schematic diagram of the main hardware topology structure between the field side and the smart device according to an embodiment of the present application; Figure 4 is a schematic diagram of the main software architecture of the battery swap station according to an embodiment of the present application; Figure 5 is a schematic diagram of the main software architecture of the vehicle side according to an embodiment of the present application.

[0150] As shown in Figure 3, the battery swap station can deploy sensor A and sensor B. The data collected by sensor A and sensor B is used to obtain the vehicle trajectory planning results through a high-performance computing platform and sent to the vehicle end through a wireless link.

[0151] As shown in Figure 4, the high-performance computing platform of the battery swap station can include a calibration module, a multimodal perception fusion module, an environment fusion module, a trajectory planning module, a fault monitoring module and a station-side communication module.

[0152] The multimodal perception fusion module performs lidar target detection based on the raw point cloud collected by the n lidars installed at the battery swap station to obtain voxelized features. It also performs camera target detection based on the raw image data collected by the n cameras installed at the battery swap station to obtain camera features. The voxelized features and camera features are fused through a neural network and post-processed to obtain the field target perception results. The field target perception results can include the perception of dynamic targets in the environment, the occupancy of static obstacles, and the positioning of the vehicle to be swapped.

[0153] The environment fusion module performs environment fusion based on the environment map and the field target perception results to obtain the field environment fusion results. The field environment fusion results can include the positioning results of dynamic targets and the vehicle to be replaced.

[0154] The trajectory planning module performs trajectory planning based on the field environment fusion results, the vehicle's functional state machine and vehicle status information, and the vehicle's scheduling information, obtaining a trajectory planning result. The trajectory planning result may include the planned trajectory and real-time position of the vehicle to be swapped.

[0155] The station-side communication module includes an information security SDK, depot time synchronization, and a link protocol layer. The station-side communication module connects to the vehicle via redundant wireless links. These redundant wireless links can be based on protocols such as Wi-Fi, 4G, and V2X. Depot time synchronization synchronizes the time between the battery swap station and the vehicle.

[0156] The fault monitoring module is used to diagnose faults in sensors, hardware, algorithms, and architectures of high-performance computing platforms.

[0157] The calibration module is used to calibrate sensors (cameras, lidars, etc.).

[0158] As shown in Figure 5, the vehicle-side autonomous driving domain controller (ADC) includes a vehicle-side communication module, control, punching, compensation algorithm module, and odometer.

[0159] The vehicle-side communication module includes an information security SDK, depot time synchronization, and a link protocol layer. The vehicle-side communication module communicates with the station-side communication module via redundant communication links. It obtains trajectory planning results from the battery swap station and sends them to the control, punching, and compensation algorithm modules. It also obtains vehicle scheduling information, and sends the vehicle's functional state machine and status information to the battery swap station. The trajectory planning results can include the planned trajectory and real-time position of the vehicle to be swapped.

[0160] The odometer performs relative positioning of the vehicle based on the vehicle's IMU and wheel speed meter data to obtain the vehicle's odometer fusion positioning data.

[0161] The control, blanking, and compensation algorithm modules include a safety and security decision processing module and a control module. The vehicle is controlled based on trajectory planning results, safety and security decision results, and odometer fusion positioning data.

[0162] The vehicle's driving control can be fed back to the battery swap station via the vehicle-side communication module, including the vehicle's functional state machine and status information, through the business logic module (such as the functional state machine). Simultaneously, fault diagnosis instructions from the station can be sent to the business logic module via the vehicle-side communication module. The business logic module can also provide display interaction based on a touchscreen interactive device (CDC).

[0163] The vehicle's driving control instructions can send control signals to the vehicle's lateral control interface and longitudinal control interface through the control instruction output. The lateral control interface is connected to the VCU (electronic power steering system), and the longitudinal control interface is connected to the EPS (electronic power steering system).

[0164] The VCU and EPS are connected to the vehicle-side communication module through the BGW (central gateway). The CDC is also connected to the vehicle-side communication module.

[0165] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present application.

[0166] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0167] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store a program for executing the autonomous driving method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned autonomous driving method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.

[0168] Furthermore, the present application provides a field-side server, which may include at least one processor; and a memory communicatively connected to the at least one processor; wherein a computer program is stored in the memory, and when the computer program is executed by at least one processor, the method described in any of the above embodiments is implemented.

[0169] In some embodiments of the present application, the field-side server is connected to a plurality of sensors set up at the field side, and the sensors are used to sense information. The sensors are communicatively connected to any type of processor of the field-side server.

[0170] In one embodiment, the memory and processor of the farm server are communicatively connected via a bus.

[0171] Furthermore, the present application provides an intelligent device, which may include at least one processor and a memory in communication with the at least one processor. The memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The intelligent device described in the present application may include a driving device, a smart car, a robot, or other devices.

[0172] In some embodiments of the present application, the smart device further includes at least one sensor for sensing information. The sensor is communicatively coupled to any type of processor in the smart device. Optionally, the smart device further includes an autonomous driving system for guiding the smart device to drive autonomously or provide assisted driving. The processor communicates with the sensor and / or the autonomous driving system to perform the method described in any of the above embodiments.

[0173] In one embodiment, the memory and the processor of the smart device are communicatively connected via a bus.

[0174] Furthermore, the present application also includes an autonomous driving system, which includes a field-side server in the field-side server embodiment and an intelligent device in the intelligent device embodiment. The field-side server and the intelligent device are communicatively connected.

[0175] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present application, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.

[0176] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principles of this application. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of this application.

[0177] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, and based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0178] The user personal information processed in this application will vary depending on the specific product / service scenario and will be based on the specific scenario in which the user uses the product / service. This may involve the user's account information, device information, driving information, vehicle information, or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0179] This application attaches great importance to the security of user personal information and has taken reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

[0180] Thus far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. An automatic driving method, characterized in that: The method is applied to a field end, and includes: Based on data collected by at least two types of sensors provided at the field end, multimodal perception fusion is performed to obtain a field end target perception result; wherein the field end target perception result includes an intelligent device for which trajectory planning is to be performed; and the intelligent device is communicatively connected to the field end; Performing environmental fusion based on the field-side target perception result and the environmental map to obtain a field-side environmental fusion result; Performing trajectory planning on the smart device according to the field-side environment fusion result to obtain a trajectory planning result of the smart device; The trajectory planning result is sent to the smart device, so that the smart device can realize the automatic driving function according to the trajectory planning result.

2. The automatic driving method according to claim 1, wherein: The step of performing trajectory planning on the smart device according to the field-side environment fusion result to obtain the trajectory planning result of the smart device includes: Obtaining status data and scheduling data of the smart device; According to the state data, the scheduling data and the field environment fusion result, the trajectory of the smart device is planned to obtain the trajectory planning result.

3. The automatic driving method according to claim 2, wherein: The obtaining of the trajectory planning result includes: Obtain the planned trajectory and real-time pose of the smart device.

4. The automatic driving method according to claim 1, wherein: The method further comprises: Achieving communication connection between the field terminal and the smart device via a wireless link; The wireless link has time synchronization capability.

5. The automatic driving method according to claim 4, characterized in that: The wireless link is a redundant wireless link; the method further includes: The redundant wireless links are implemented according to a variety of different types of wireless communication protocols.

6. The automatic driving method according to claim 1, wherein: The multimodal fusion is implemented by a multimodal fusion module; the environmental fusion is implemented by an environmental fusion module; the trajectory planning is implemented by a trajectory planning module; and the method further includes: Performing fault monitoring on the sensor, the multimodal fusion module, the environment fusion module, and the trajectory planning module; When at least one of the sensor, the multimodal fusion module, the environment fusion module and the trajectory planning module fails, a fault self-correction mechanism responds and fault information corresponding to the fault is saved.

7. The automatic driving method according to claim 1, wherein: The sensor includes a first sensor and a second sensor, and the number of the first sensor and the number of the second sensor are both multiple; The performing multimodal perception fusion based on data collected by at least two types of sensors provided at the field end to obtain a field end target perception result includes: Acquire a first perception feature based on data collected by the first sensor; acquiring a second perception feature based on data collected by the second sensor; Performing feature fusion based on the first perception feature and the second perception feature to obtain a perception fusion feature; The field-side target perception result is obtained according to the perception fusion feature.

8. The automatic driving method according to claim 7, characterized in that: The obtaining of the field-side target perception result includes: According to the perception fusion feature, the dynamic target perception result of the environment, the static obstacle occupancy result and the positioning result of the smart device are obtained.

9. The automatic driving method according to claim 1, wherein: The method further comprises obtaining the environment map according to the following steps: Based on the data collected by the sensor, acquiring the field environment data of the field end; Based on the field environment data, acquiring static obstacle occupancy data; The environment map is obtained according to the static obstacle occupancy data.

10. The automatic driving method according to claim 9, characterized in that: The method further comprises: updating the environment map according to the return flow data; The reflow data is data collected by sensors of the field end or the smart device that triggers reflow based on preset rules.

11. The automatic driving method according to claims 1 to 10, characterized in that: The method further comprises: Performing on-site calibration or online calibration on the sensor to obtain an external parameter calibration result of the sensor; The external parameter calibration result is used for the multimodal perception fusion and / or the environment fusion.

12. An automatic driving method, characterized in that: The method is applied to a smart device and includes: Obtaining a trajectory planning result of the smart device sent by a field end; wherein the field end is in communication with the smart device; Controlling the driving of the smart device according to the trajectory planning result and the safety guard decision result of the smart device to realize the automatic driving function of the smart device; The safety guard decision result is a result obtained after the smart device performs safety control evaluation and verification on the trajectory planning result.

13. The automatic driving method according to claim 12, wherein: The step of controlling the driving of the smart device according to the trajectory planning result and the safety guard decision result of the smart device includes: Performing priority judgment based on the trajectory planning result and the security guard decision result of the smart device to obtain a priority judgment result; Driving control of the smart device is performed based on the priority judgment result and the odometer fusion positioning data of the smart device.

14. The automatic driving method according to claim 13, wherein: The step of controlling the driving of the smart device based on the priority determination result and the odometer-fused positioning data of the smart device includes: When the priority of the trajectory planning result is higher than the priority of the safety guard decision result, controlling the driving of the smart device according to the trajectory planning result and the odometer fused positioning data; When the priority of the safety guard decision result is higher than the priority of the trajectory planning result, the driving of the smart device is controlled according to the safety guard decision result and the odometer fusion positioning data.

15. The automatic driving method according to claim 14, wherein: The smart device is connected to the field terminal via a wireless link, and the wireless link has a time synchronization capability; The step of controlling the driving of the smart device according to the trajectory planning result and the odometer-fused positioning data includes: Time synchronization of the trajectory planning result via the wireless link; Based on the time-synchronized trajectory planning result and the odometer fusion positioning data, the intelligent The device performs driving control.

16. The automatic driving method according to claim 15, characterized in that: The trajectory planning result includes the planned trajectory and real-time pose of the smart device; The method of controlling the driving of the smart device based on the time-synchronized trajectory planning result and the odometer fusion positioning data includes: Performing motion compensation on the time-synchronized real-time posture according to the odometer fusion positioning data to obtain a compensated real-time posture; The intelligent device is controlled to travel based on the compensated real-time posture and the time-synchronized planned trajectory.

17. The automatic driving method according to claim 15 or 16, characterized in that: The wireless link is a redundant wireless link; the method further includes: The redundant wireless links are implemented according to a variety of different types of wireless communication protocols.

18. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the autonomous driving method according to any one of claims 1 to 11 or the autonomous driving method according to any one of claims 12 to 17.

19. A field-side server, characterized in that: The field-side server includes: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the automatic driving method according to any one of claims 1 to 11 is implemented.

20. A smart device, characterized in that: The smart device includes: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, a computer program is stored in the memory, and when the computer program is executed by the at least one processor, the automatic driving method according to any one of claims 12 to 17 is implemented.

21. An automatic driving system, characterized in that: The autonomous driving system includes the field server according to claim 19 and the smart device according to claim 20, and the field server is communicatively connected to the smart device.

Images