Parallel driving assistance method, device, parallel driving system and electronic equipment

Abstract

The application discloses a parallel driving assistance method and device, a parallel driving system and electronic equipment. The parallel driving assistance method comprises the following steps: receiving self-vehicle data uploaded by an automatic driving vehicle, and determining a current state of a cockpit end, wherein the self-vehicle data comprises self-vehicle state data; if the cockpit end is in a monitoring state, obtaining a control instruction of a simulation control device according to the self-vehicle state data; and reversely controlling the simulation control device according to the control instruction of the simulation control device, so that the state of the simulation control device is synchronized with the state of the vehicle. The technical scheme of the application synchronizes the state of the simulation control device with the state of the vehicle in the monitoring state of the cockpit end, thereby avoiding state mutation of the controlled vehicle under the action of a control vehicle instruction when the cockpit switches from the monitoring state to a remote control state, and causing a safety hazard.

Description

Technical Field

[0001] The present application relates to the field of parallel driving technology, and in particular to a parallel driving assistance method, device, parallel driving system and electronic equipment. Background Art

[0002] The development of autonomous driving technology will gradually move toward unmanned operation. Remote control can play a key supporting role in the transition from Level 4 to Level 5 autonomous driving. Remote control allows an operator to control a vehicle outside their field of view via wired or wireless networks using simulated control devices such as steering wheel simulators, pedal simulators, and display screens. The simulated control devices are connected to the controlled vehicle via a local or wide area network, allowing the control distance to transcend spatial limitations.

[0003] like Figure 1 As shown, it is a schematic diagram of a parallel driving system for implementing remote control. The parallel driving system is an auxiliary control system for remotely controlling vehicles. Parallel driving refers to a vehicle traveling on the road, which collects video data through the vehicle-side camera and obtains vehicle status data through the remote driving controller, and sends the vehicle status data and video data to the remote control system through a wireless network (such as 5G base station, 5G core network, etc.). The remote control system displays the returned video, vehicle status and other data on the monitoring screen, so that the driver can observe the monitoring screen to understand the vehicle's driving status, and control the controlled vehicle through the analog control device, and perform a series of remote action controls on the vehicle such as acceleration, deceleration, and turning.

[0004] The study found that during the remote control process, the status of the analog control device will affect the safety of vehicle remote control. Summary of the Invention

[0005] The embodiments of the present application provide a parallel driving assistance method, device, parallel driving system and electronic equipment to assist the cockpit in remote control and improve the safety of vehicle remote control.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a parallel driving assistance method, which is executed by a cockpit end, comprising:

[0008] Receive vehicle data uploaded by the autonomous vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle status data;

[0009] If the cockpit is in a monitoring state, a control instruction of the simulation control device is obtained according to the vehicle state data;

[0010] The simulation control device is reversely controlled according to a control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle.

[0011] Optionally, the vehicle state data includes vehicle-side steering wheel angle information, and obtaining a control instruction of the simulation control device according to the vehicle state data includes:

[0012] Acquire cockpit-side steering wheel angle information corresponding to the vehicle-side steering wheel angle information, and generate a steering wheel simulator control instruction corresponding to the cockpit-side steering wheel angle information;

[0013] The reversely controlling the simulation control device according to the control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle includes:

[0014] The steering wheel simulator is reversely controlled according to the steering wheel simulator control instruction so that the steering wheel simulator is synchronized with the steering wheel angle of the vehicle.

[0015] Optionally, the vehicle status data includes vehicle-side indicator light switch information, and obtaining a control instruction of the simulation control device according to the vehicle status data includes:

[0016] Generate cockpit indicator light control instructions according to the vehicle-side indicator light switch information;

[0017] The reversely controlling the simulation control device according to the control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle includes:

[0018] The cockpit indicator light switch is reversely controlled according to the cockpit indicator light control instruction, so that the on / off state of the cockpit indicator light is synchronized with the on / off state of the vehicle-end indicator light.

[0019] Optionally, when determining the current state of the cockpit terminal, the method further includes:

[0020] If the cockpit side switches from the remote control state to the monitoring state, the control simulation control device is controlled to maintain the current state until new vehicle state data is received.

[0021] Optionally, the vehicle status data includes vehicle position information, and the vehicle data also includes ADAS data. When receiving the vehicle data uploaded by the vehicle, the method further includes:

[0022] Obtaining a vehicle driving state model based on the vehicle state data, and obtaining a traffic participant model based on the ADAS data;

[0023] Acquire corresponding high-precision map data according to the vehicle position information;

[0024] The vehicle driving state model and traffic participant model are rendered on the high-precision map data according to the vehicle state data and the ADAS data to obtain and display a three-dimensional motion model centered on the vehicle.

[0025] Optionally, the method further includes:

[0026] Determining whether a remote takeover request from the vehicle is received;

[0027] When the remote takeover request is received, a remote control task is generated and displayed according to the remote takeover request.

[0028] Optionally, the method further includes:

[0029] Obtaining downlink delay statistics, where the downlink delay statistics include the downlink delay from a vehicle control command in the cockpit to the autonomous vehicle executing the vehicle control command;

[0030] Control the cockpit end to display the downlink delay statistics.

[0031] In a second aspect, an embodiment of the present application provides a parallel driving assistance device, which is applied to a management server, and the device includes:

[0032] A receiving and state determination unit, configured to receive vehicle data uploaded by the autonomous driving vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle state data;

[0033] a control instruction generating unit, configured to obtain a control instruction of the simulation control device according to the vehicle status data if the cockpit terminal is in a monitoring state;

[0034] The state synchronization control unit is used to reversely control the simulation control device according to the control instructions of the simulation control device, so that the state of the simulation control device is synchronized with the vehicle state.

[0035] In a third aspect, an embodiment of the present application provides a parallel driving system, including a vehicle side, a service side, and a cockpit side;

[0036] The vehicle side is used to upload the vehicle data to the service side, wherein the vehicle data includes the vehicle status data;

[0037] The server is used to send the vehicle data to the cockpit;

[0038] The cockpit end is used to execute the parallel driving assistance method.

[0039] Optionally, the vehicle-side is configured to generate a remote takeover request based on a fault detection result of a vehicle-side fault self-diagnosis program; and / or obtain a decision instruction generated by a vehicle-side autonomous driving program and generate a remote takeover request based on the reliability of the decision instruction; and / or generate a remote takeover request when the vehicle-side is in a distressed state due to an unexpected parking during autonomous driving; and upload the remote takeover request to the server-side;

[0040] The server is configured to send the remote takeover request to the cockpit;

[0041] The cockpit end is used to receive the remote takeover request, generate and display a remote control task according to the remote takeover request.

[0042] Optionally, the cockpit side is used to generate a remote vehicle control instruction based on the displayed remote control task, the three-dimensional motion model centered on the vehicle, and the vehicle-side monitoring video, and send the remote vehicle control instruction to the server side;

[0043] The server is used to send the remote vehicle control command to the vehicle;

[0044] The vehicle end is used to receive the remote vehicle control command and switch to the remote driving mode.

[0045] In a fourth aspect, an embodiment of the present application provides an electronic device comprising: a processor; and a memory arranged to store computer-executable instructions, which, when executed, enable the processor to execute a parallel driving assistance method.

[0046] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device performs a parallel driving assistance method.

[0047] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0048] The embodiment of the present application first receives the self-driving vehicle data uploaded by the autonomous vehicle and determines the current state of the cockpit end, wherein the self-driving vehicle data includes self-vehicle status data; when the cockpit end is in the monitoring state, the control instructions of the simulation control device are obtained according to the self-vehicle status data; the simulation control device is reversely controlled according to the control instructions of the simulation control device, so that the state of the simulation control device is synchronized with the vehicle state. When the cockpit end is in the monitoring state, the present application synchronizes the state of the simulation control device with the vehicle state to avoid the controlled vehicle from suddenly changing its state under the control of the vehicle control instruction when the cockpit switches from the monitoring state to the remote control state, thereby preventing a safety hazard. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 A schematic diagram of the interaction between the vehicle control system and the remote control system;

[0051] Figure 2 This is a flow chart of a parallel driving assistance method in an embodiment of the present application;

[0052] Figure 3 This is a schematic structural diagram of a parallel driving assistance device in an embodiment of the present application;

[0053] Figure 4 This is a structural diagram of a parallel driving system in an embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of information interaction of a parallel driving system in an embodiment of the present application;

[0055] Figure 6 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0056] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0057] The embodiment of the present application provides a parallel driving assistance method, such as Figure 2 As shown, a flow chart of a parallel driving assistance method in an embodiment of the present application is provided, and the method at least includes the following steps S210 to S230:

[0058] Step S210: receiving the self-driving vehicle data uploaded by the autonomous vehicle and determining the current state of the cockpit terminal, wherein the self-driving vehicle data includes the self-driving vehicle state data.

[0059] The parallel driving assistance method of this application is executed by the cockpit. The autonomous vehicle reports its own vehicle data to the server in real time. The server processes the self-driving data and sends it to the corresponding cockpit. The cockpit receives the self-driving data sent by the server. The self-driving data includes self-driving status data, which includes attribute information and operational information. Attribute information includes, for example, vehicle identification, vehicle type, and vehicle dimensions. Operational information includes, for example, vehicle positioning data, vehicle dynamics data, and vehicle operation data.

[0060] In this application, the cockpit's operating status can include monitoring and remote control. Monitoring refers to when the remote driver is not controlling the autonomous vehicle, while remote control refers to when the remote driver issues vehicle control commands from the cockpit, switching the autonomous vehicle from autonomous mode to parallel driving mode. The cockpit's current status can be determined based on the operating status indicator.

[0061] Step S220: If the cockpit terminal is in the monitoring state, a control instruction of the simulation control device is obtained according to the vehicle state data.

[0062] Step S230 , reversely controlling the simulation control device according to the control instruction of the simulation control device, so that the state of the simulation control device is synchronized with the state of the vehicle.

[0063] The study found that when the cockpit is in a monitoring state, if the state of the simulation control device is not synchronized with the vehicle state, for example, the steering wheel simulator's angle is inconsistent with the vehicle-side steering wheel angle, when the cockpit switches from a monitoring state to a remote control state, the steering control command issued by the steering wheel simulator will act on the controlled vehicle, causing the vehicle-side steering wheel angle to jump, and the controlled vehicle is prone to drive in a dangerous direction.

[0064] Based on this, the present application generates control instructions for a simulation control device according to the vehicle status data when the cockpit is in a monitoring state, and reversely controls the simulation control device based on the control instructions of the simulation control device. For example, control instructions for a steering wheel simulator are generated according to the vehicle-end steering wheel angle information included in the vehicle status data, and the steering wheel simulator is reversely controlled using the control instructions to make the steering wheel simulator's angle consistent with the vehicle-end steering wheel's angle, thereby avoiding sudden changes in the vehicle-end steering wheel angle when the cockpit switches from a monitoring state to a remote control state, and improving the safety of remote operation.

[0065] It can be understood that the synchronous control of the state of the simulation control device and the vehicle-side state in this application includes at least the following two types of state synchronization, one is the synchronization of the steering wheel angle, and the other is the synchronization of the indicator light switch state.

[0066] When the vehicle status data includes the vehicle-side steering wheel angle information, the steering wheel angle synchronization should be performed. The specific steps are as follows:

[0067] First, the cockpit end obtains the cockpit end steering wheel angle information corresponding to the vehicle end steering wheel angle information; for example, when the server end receives the vehicle status data reported by the vehicle, it converts the vehicle end steering wheel angle information into the cockpit end steering wheel angle information according to the preset conversion strategy. The preset conversion strategy can be a conversion according to a ratio or a conversion according to an angle conversion model. The server end sends the converted cockpit end steering wheel angle information to the cockpit end. The cockpit end obtains the cockpit end steering wheel angle information by parsing the message sent by the server end, and the cockpit end generates a steering wheel simulator control instruction corresponding to the cockpit end steering wheel angle information.

[0068] Secondly, the steering angle of the steering wheel simulator is reversely controlled according to the steering wheel simulator control instruction, so that the steering angle of the steering wheel simulator is synchronized with the steering angle of the vehicle-end steering wheel.

[0069] In this way, the steering wheel simulator and the vehicle-side steering wheel angle are synchronized through the above steps.

[0070] When the vehicle status data includes the vehicle-side indicator light switch information, the indicator light switch status synchronization should be performed. The specific steps are as follows:

[0071] A cockpit indicator light control instruction is generated according to the vehicle-end indicator light switch information, and the cockpit indicator light switch is reversely controlled according to the cockpit indicator light control instruction, so that the switch state of the cockpit indicator light is synchronized with the switch state of the vehicle-end indicator light.

[0072] In this way, the switching state of the indicator light simulator and the vehicle-side indicator light are synchronized through the above steps.

[0073] It should be noted that, considering that the vehicle-side speed and acceleration are continuously changing values, even if the cockpit and vehicle-side speeds and accelerations are not synchronized, the instantaneous changes in the cockpit's speed and acceleration will not significantly affect the instantaneous values ​​of the vehicle-side speed and acceleration, nor will they affect the safety of remote control on the vehicle side. Based on this, in an optional embodiment of the present application, when the vehicle-side state data includes data such as vehicle-side speed and vehicle-side acceleration, it can be determined based on demand whether to make the cockpit pedal simulator and the vehicle-side pedal be in the same state.

[0074] In the above-mentioned embodiment of the present application, when the cockpit is in the monitoring state, the state of the simulation control device and the vehicle state are synchronously controlled to avoid the controlled vehicle from suddenly changing state under the action of the vehicle control command when the cockpit switches from the monitoring state to the remote control state, thereby preventing safety hazards.

[0075] In one embodiment of the present application, in order to avoid sudden changes in the vehicle state, when the cockpit end switches from the remote control state to the monitoring state, the control simulation control device is controlled to maintain the current state until new vehicle state data is received.

[0076] Currently, remote control primarily involves the remote driver observing information on a central control screen or display in the cockpit, and then using their experience to determine whether to take over remote control. Currently, the information displayed on the central control screen or display is generally just the vehicle's status and audio and video data captured by the vehicle's cameras. This information display method fails to effectively provide a clear understanding of the vehicle's surroundings, hindering driver decision-making.

[0077] In response to the above-mentioned problems existing in the prior art, in one embodiment of the present application, the vehicle data includes vehicle status data, audio and video data, and ADAS (Advanced Driving Assistance System) data. ADAS data refers to the surrounding information perceived by the autonomous driving vehicle through sensors. For example, the surrounding environment is collected through sensors such as lidar, millimeter-wave radar, and cameras to obtain perception information about the surrounding environment of the autonomous driving vehicle, and the various types of perceived information identified are fused and calculated to obtain traffic participant data.

[0078] Upon receiving the vehicle data uploaded by the vehicle, the parallel driving assistance method further includes:

[0079] A vehicle driving state model is obtained based on the vehicle state data, and a traffic participant model is obtained based on the ADAS data included in the vehicle data; corresponding high-precision map data is obtained based on the vehicle position information, where the high-precision map data is preferably local high-precision map data, such as high-precision map data within a certain range centered on the vehicle position; the vehicle driving state model and the traffic participant model are rendered on the high-precision map data based on the vehicle state data and the ADAS data, and a three-dimensional motion model centered on the vehicle is obtained and displayed, effectively assisting the remote driver in making control decisions.

[0080] In this embodiment, the audio and video data are used to generate voice data of the three-dimensional motion model on the one hand, and on the other hand, are used to live broadcast the vehicle's driving conditions on the cockpit side, further assisting the remote driver in making control decisions.

[0081] Compared to the existing technology, the embodiment of the present application combines high-precision maps to dynamically display the environment around the autonomous driving vehicle. The high-precision map data not only contains high-precision coordinates, but also accurate road shapes, and the slope, curvature, heading, elevation, and roll data of each lane are also included; in addition, the type of signs on each lane, the color of the lane lines, the road divider, the arrows and text on the road signs will all be presented in the high-precision map, and the vehicle's driving status model and traffic participant model rendered on the high-precision map can intuitively display the situation of traffic participants near the autonomous driving vehicle. Therefore, the embodiment of the present application can provide strong support for remote control by remote drivers based on the rich road structured information and traffic participant status information provided in the high-precision map data.

[0082] In practice, autonomous vehicles perceive their surroundings through sensors like lidar, millimeter-wave radar, and cameras. This information is then fused and calculated to generate traffic participant data, also known as ADAS data. Its attributes include the type, location, size, direction, and speed of traffic participants. The vehicle-side parallel driving program transmits this ADAS data to the cockpit, which loads a local high-precision map. The ADAS data transmitted by the autonomous vehicle is then rendered onto the high-precision map as a model, with its attributes assigned to the corresponding functions. Simultaneously, vehicle status data, such as its location, size, direction, and speed, is also rendered onto the high-precision map, forming a three-dimensional motion model with the autonomous vehicle as the bird's-eye view. This 3D motion model allows for intuitive observation of real-time dynamic information about traffic participants around the autonomous vehicle, assisting the remote driver in making better control decisions.

[0083] In one embodiment of the present application, the parallel driving assistance method further includes:

[0084] Determining whether a remote takeover request from the vehicle is received;

[0085] When the remote takeover request is received, a remote control task is generated and displayed according to the remote takeover request, and the display method includes sound reminder, visual-based graphic display, etc.

[0086] Taking one application scenario of this embodiment as an example, the autonomous vehicle's fault self-diagnosis program detects vehicle-side hardware and software faults and generates a remote control request based on the detected fault information. Fault detection includes LiDAR faults, millimeter-wave radar faults, camera faults, RTK faults, and so on. When these hardware devices or associated software programs fail, autonomous driving may not be completely impossible, but its autonomous driving performance may be affected. In this case, the autonomous vehicle generates a remote takeover request and reports it to the server, which forwards it to the cockpit.

[0087] Furthermore, after the autonomous driving program of the autonomous driving vehicle generates an autonomous driving decision instruction, it also determines the reliability of its autonomous driving decision. For example, when the reliability of the autonomous driving decision is higher than 95% but lower than 97%, the autonomous driving program will generate and report a remote takeover request. At this time, the autonomous driving safety factor is low and remote monitoring assistance is required. When the autonomous driving is in a safe state (for example, the reliability of the autonomous driving decision is greater than 97%), the parallel driving system will prompt "Entered a safe state" and the driver can exit and process other tasks.

[0088] Furthermore, if the vehicle's autonomous driving program encounters a situation outside of its ODD (Operational Design Domain) or is unable to execute autonomous driving decisions due to hardware or software failures, the autonomous vehicle stops driving and generates a remote takeover request to the server, which then forwards it to the corresponding cockpit. At this point, the autonomous vehicle is in distress, and upon receiving the remote takeover request, the cockpit remotely controls the vehicle and assists in its escape.

[0089] In some cases, some embodiments of the present application can set a remote takeover strategy and send the remote takeover strategy to the autonomous driving vehicle, so that the autonomous driving vehicle determines whether to generate a remote takeover request based on the remote takeover strategy. When it is determined that a remote takeover request needs to be generated, the remote takeover request is forwarded to the cockpit end through the server end. When the cockpit end receives the remote takeover request, the remote takeover request is added to the task list to be processed as a task event. During the adding process, the remote takeover request can be displayed accordingly according to the priority of the remote takeover request.

[0090] In some embodiments, the parallel driving assistance method further includes:

[0091] Obtaining downlink delay statistics, where the downlink delay statistics include the downlink delay from a vehicle control command in the cockpit to the autonomous vehicle executing the vehicle control command;

[0092] Control the cockpit end to display the downlink delay statistics.

[0093] For example, during the operation of the cockpit program, the server program and the vehicle-side parallel driving program, they collect the issuance, arrival and processing time of each instruction, and send the collected data to the server-side delay monitoring module. The delay monitoring module collects the delay data and makes summary statistics, which at least includes downlink delay statistics. The statistical results are saved and transmitted to the cockpit program in real time, and displayed on the central control screen and display screen of the cockpit.

[0094] Based on the above embodiments, the parallel driving assistance method of the present application maintains real-time synchronization between the simulation control device on the cockpit side and the vehicle side status during the monitoring stage, thereby improving the safety of remote control of the vehicle, and provides a three-dimensional motion model, remote takeover request, and downlink delay statistics to assist the remote driver in making remote control decisions, and realizes the ability of autonomous driving vehicles to fail, escape from trouble, and seek help, thereby increasing the probability of remote control assistance to escape from trouble.

[0095] The parallel driving instruction processing method in the aforementioned embodiment has the same technical concept. The embodiment of the present application further provides a parallel driving assistance device 300, such as Figure 3 As shown, a schematic diagram of the structure of a parallel driving assistance device in an embodiment of the present application is provided. The parallel driving assistance device 300 is applied to the cockpit end and includes: a receiving and state determination unit 310, a control instruction generation unit 320 and a state synchronization control unit 330, wherein:

[0096] The receiving and state determination unit 310 is used to receive vehicle data uploaded by the autonomous driving vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle state data;

[0097] The control instruction generating unit 320 is used to obtain the control instruction of the simulation control device according to the vehicle status data if the cockpit terminal is in the monitoring state;

[0098] The state synchronization control unit 330 is used to reversely control the simulation control device according to the control instruction of the simulation control device, so that the state of the simulation control device is synchronized with the vehicle state.

[0099] In one embodiment of the present application, the vehicle state data includes vehicle-side steering wheel angle information, and the control instruction generation unit 320 is configured to obtain cockpit-side steering wheel angle information corresponding to the vehicle-side steering wheel angle information, and generate a steering wheel simulator control instruction corresponding to the cockpit-side steering wheel angle information;

[0100] Correspondingly, the state synchronization control unit 330 is used to reversely control the steering angle of the steering wheel simulator according to the steering wheel simulator control instruction, so that the steering angle of the steering wheel simulator is synchronized with the steering angle of the vehicle-end steering wheel.

[0101] In one embodiment of the present application, the vehicle status data includes vehicle-side indicator light switch information, and the control instruction generation unit 320 is configured to generate a cockpit-side indicator light control instruction based on the vehicle-side indicator light switch information;

[0102] Correspondingly, the state synchronization control unit 330 is used to reversely control the cockpit indicator light switch according to the cockpit indicator light control instruction, so that the switch state of the cockpit indicator light is synchronized with the switch state of the vehicle-end indicator light.

[0103] In one embodiment of the present application, the parallel driving assistance device 300 further includes a first auxiliary control unit;

[0104] The first auxiliary control unit is used to control the simulation control device to maintain the current state until new vehicle state data is received when determining the current state of the cockpit end if the cockpit end switches from the remote control state to the monitoring state.

[0105] In one embodiment of the present application, the ego vehicle state data includes ego vehicle position information, the ego vehicle data also includes ADAS data, and the parallel driving assistance device 300 further includes a three-dimensional motion model processing unit;

[0106] The three-dimensional motion model processing unit is used to obtain the vehicle's driving state model based on the vehicle's state data, and to obtain the traffic participant model based on the ADAS data; to obtain the corresponding high-precision map data based on the vehicle's position information; and to render the vehicle's driving state model and the traffic participant model on the high-precision map data based on the vehicle's state data and the ADAS data to obtain and display the three-dimensional motion model centered on the vehicle.

[0107] In one embodiment of the present application, the parallel driving assistance device 300 further includes a second auxiliary control unit and a third auxiliary control unit;

[0108] The second auxiliary control unit is used to determine whether a remote takeover request from the vehicle is received; when the remote takeover request is received, a remote control task is generated and displayed according to the remote takeover request.

[0109] The third auxiliary control unit is used to obtain downlink delay statistics, which include the downlink delay from the vehicle control command in the cockpit to the automatic driving vehicle executing the vehicle control command; and control the cockpit to display the downlink delay statistics.

[0110] It can be understood that the above-mentioned parallel driving assistance device 300 can implement the various steps of the parallel driving assistance method provided in the aforementioned embodiment. The relevant explanations about the parallel driving assistance method are applicable to the parallel driving assistance device 300 and will not be repeated here.

[0111] The embodiment of the present application also provides a parallel driving system 400, such as Figure 4 , a schematic structural diagram of a parallel driving system according to an embodiment of the present application is provided. The parallel driving system 400 includes: a vehicle end 410, a service end 420, and a cockpit end 430, wherein:

[0112] The vehicle side 410 is used to upload the vehicle data to the server side, where the vehicle data includes the vehicle status data.

[0113] In this embodiment, after the vehicle side establishes a network connection with the server side, the vehicle side automatic driving monitoring program runs and collects data from the vehicle and uploads it to the server side.

[0114] The server 420 is used to send the vehicle data to the cockpit.

[0115] The cockpit end 430 is used to execute the parallel driving assistance method as described in the aforementioned embodiment.

[0116] In some embodiments, the cockpit end is also used to select the target vehicle end, and generate a live broadcast instruction of the target vehicle end and send it to the server end 420. The server end 420 sends the live broadcast instruction to the target vehicle, and the target vehicle end 510 starts audio and video live broadcast according to the live broadcast instruction. At this time, the central control screen or display screen of the cockpit end displays the monitoring video of the vehicle end, and the headphones of the cockpit end play the ambient sound collected by the vehicle end.

[0117] In some embodiments, the vehicle-side 410 is used to generate a remote takeover request based on the fault detection results of the vehicle-side fault self-detection program. The vehicle-side fault self-detection program detects vehicle-side hardware and software faults. Fault detection includes lidar faults, millimeter-wave radar faults, camera faults, RTK faults, etc. When these hardware devices or associated software programs fail, they may not be completely unable to drive automatically, but their autonomous driving effects may be affected. At this time, the autonomous driving vehicle will generate a remote takeover request and report it to the server.

[0118] And / or obtain the decision instructions generated by the vehicle-side autonomous driving program, and generate a remote takeover request based on the reliability of the decision instructions. For example, when the reliability of the autonomous driving decision is higher than 95% but lower than 97%, the autonomous driving program will generate a remote takeover request and report it.

[0119] And / or when the vehicle is in a distressed state due to an unexpected parking during the autonomous driving process, a remote takeover request is generated; the remote takeover request is uploaded to the server. For example, when the vehicle-side autonomous driving program encounters a situation outside the ODD or is unable to execute the autonomous driving decision due to software or hardware failure, the autonomous driving vehicle stops driving. At this time, the autonomous driving vehicle is in a distressed state and a remote takeover request is generated.

[0120] The server 420 is used to send the remote takeover request to the cockpit end;

[0121] The cockpit end 430 is used to receive the remote takeover request, generate and display a remote control task according to the remote takeover request.

[0122] In some embodiments, the cockpit end 430 is used to generate a remote vehicle control instruction based on the displayed remote control task, the three-dimensional motion model centered on the vehicle, and the vehicle-side monitoring video, and send the remote vehicle control instruction to the server end;

[0123] The server 420 is used to send the remote vehicle control command to the vehicle;

[0124] The vehicle end 410 is used to receive the remote vehicle control command and switch to the remote driving mode.

[0125] It should be noted that Figure 4 The structural composition of the parallel driving system is only shown as an example. In an optional embodiment, the parallel driving system includes multiple vehicle ends and multiple cockpit ends. For the service end, it can include one or more.

[0126] To explain the remote operation assistance method for the automatic driving cabin in detail, Figure 5 Provide detailed explanation.

[0127] like Figure 5 As shown, the parallel driving system of the embodiment of the present application includes a vehicle side, a service side and a cockpit side. The cockpit side includes simulator hardware, a parallel driving center and a client side. The simulator hardware includes, for example, cockpit shell, seats, steering wheel simulators, pedal simulators, gear remote sensors and other simulation control devices, central control screens, display screens, etc. The parallel driving center includes, for example, a monitoring module, an audio and video live broadcast module, a task management module, a login authentication module, a communication management module, etc. The client side includes a connection module and an operation module. The client side communicates with the parallel driving center through the connection module, so that the remote driver can log in to the parallel driving center through the client side to obtain vehicle control authority. The network condition of the cockpit side is a wired network with a bandwidth of 20Mbps or more. The server side includes vehicle monitoring services, remote driving services and audio and video services, etc. The vehicle side includes a parallel driving module, an automatic driving module and the vehicle's own hardware and software.

[0128] When the autonomous driving vehicle and the cockpit establish a connection with the server, the relevant programs of the autonomous driving module on the vehicle side of the autonomous driving vehicle run, collect the vehicle data and report it to the server in real time; at this time, the remote driver logs in to the parallel driving center through the client, for example, by using a mouse and keyboard to log in, or by using a Pad (tablet) to establish a connection with the parallel driving center and then log in on the Pad side. After successful login, the online vehicle list under the remote driver's authority is displayed. When the target vehicle is selected from the online vehicle list, the server is notified that the target vehicle is selected. The server notifies the target vehicle to start audio and video live broadcast, and the central control screen or display screen on the cockpit side displays the monitoring video and plays sound data.

[0129] The server forwards the vehicle data to the cockpit. The cockpit generates a three-dimensional motion model based on the vehicle status data, ADAS data and high-precision map data, and displays it through the central control screen or display screen to better assist the remote driver in completing safe driving control.

[0130] When the cockpit is in monitoring mode, the status of the simulated control device on the cockpit is synchronized with the vehicle-side status in real time. The remote driver on the cockpit determines whether to take over the autonomous vehicle based on real-time vehicle-side monitoring video, a three-dimensional motion model centered on the vehicle, and remote control tasks generated based on remote takeover requests. When the autonomous vehicle is determined to be in a dangerous state or is about to be in a dangerous state, the remote driver proactively takes over the vehicle. For example, the cockpit generates remote control commands based on the remote driver's operations. The remote control commands are forwarded to the controlled vehicle via the server, and the controlled vehicle switches from autonomous driving mode to remote driving mode. The cockpit then generates control commands and sends them to the controlled vehicle via the server. The parallel driving module on the controlled vehicle sends the control commands to the CAN bus to control the vehicle's driving.

[0131] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 6 At the hardware level, the electronic device includes a processor and memory, and optionally an internal bus and a network interface. The memory may include internal memory, such as high-speed random-access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for its services.

[0132] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0133] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0134] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a parallel driving assistance device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0135] Receive vehicle data uploaded by the autonomous vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle status data;

[0136] If the cockpit is in a monitoring state, a control instruction of the simulation control device is obtained according to the vehicle state data;

[0137] The simulation control device is reversely controlled according to a control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle.

[0138] The above application Figure 2The method performed by the parallel driving assistance device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the parallel driving assistance method described above.

[0139] The electronic device may also perform Figure 2 A method for executing a parallel driving assistance device in a vehicle and realizing a parallel driving assistance device in a vehicle Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0140] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 2 The method performed by the parallel driving assistance device in the illustrated embodiment is specifically used to perform:

[0141] Receive vehicle data uploaded by the autonomous vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle status data;

[0142] If the cockpit is in a monitoring state, a control instruction of the simulation control device is obtained according to the vehicle state data;

[0143] The simulation control device is reversely controlled according to a control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle.

[0144] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0148] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0149] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0150] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0151] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0152] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A parallel driving assistance method, characterized in that: Executed by the cockpit end, the method includes: Receive vehicle data uploaded by the autonomous vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle status data; If the cockpit is in a monitoring state, a control instruction of the simulation control device is obtained according to the vehicle state data; reversely controlling the simulation control device according to a control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle; When determining the current state of the cockpit terminal, the method further includes: If the cockpit switches from remote control state to monitoring state, the control simulation control device is controlled to maintain the current state until new vehicle state data is received; The monitoring state refers to a state in which the remote driver does not control the autonomous vehicle, and the remote control state refers to a state in which the remote driver issues a vehicle control command to the autonomous vehicle at the cockpit end, causing the autonomous vehicle to enter a parallel driving mode from an autonomous driving mode.

2. The method according to claim 1, wherein: The vehicle state data includes vehicle-side steering wheel angle information, and obtaining a control instruction of the simulation control device according to the vehicle state data includes: Acquire cockpit-side steering wheel angle information corresponding to the vehicle-side steering wheel angle information, and generate a steering wheel simulator control instruction corresponding to the cockpit-side steering wheel angle information; The reversely controlling the simulation control device according to the control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle includes: The steering wheel simulator is reversely controlled according to the steering wheel simulator control instruction so that the steering wheel simulator is synchronized with the steering wheel angle of the vehicle.

3. The method according to claim 1, wherein: The vehicle status data includes vehicle-side indicator light switch information, and obtaining a control instruction of the simulation control device according to the vehicle status data includes: Generate cockpit indicator light control instructions according to the vehicle-side indicator light switch information; The reversely controlling the simulation control device according to the control instruction of the simulation control device so that the state of the simulation control device is synchronized with the state of the vehicle includes: The cockpit indicator light switch is reversely controlled according to the cockpit indicator light control instruction, so that the on / off state of the cockpit indicator light is synchronized with the on / off state of the vehicle-end indicator light.

4. The method according to claim 1, wherein: The vehicle status data includes vehicle location information, and the vehicle data also includes ADAS data. When receiving the vehicle data uploaded by the vehicle, the method further includes: Obtaining a vehicle driving state model based on the vehicle state data, and obtaining a traffic participant model based on the ADAS data; Acquire corresponding high-precision map data according to the vehicle position information; The vehicle driving state model and traffic participant model are rendered on the high-precision map data according to the vehicle state data and the ADAS data to obtain and display a three-dimensional motion model centered on the vehicle.

5. The method according to claim 1, wherein: The method further comprises: Determining whether a remote takeover request from the vehicle is received; When the remote takeover request is received, a remote control task is generated and displayed according to the remote takeover request.

6. The method according to claim 1, wherein: The method further comprises: Obtaining downlink delay statistics, where the downlink delay statistics include the downlink delay from a vehicle control command in the cockpit to the autonomous driving vehicle executing the vehicle control command; Control the cockpit end to display the downlink delay statistics.

7. A parallel driving assistance device, characterized in that: Applied to the cockpit, the device includes: a receiving and state determination unit, configured to receive vehicle data uploaded by the autonomous driving vehicle and determine the current state of the cockpit terminal, wherein the vehicle data includes vehicle state data; a control instruction generating unit, configured to obtain a control instruction of the simulation control device according to the vehicle status data if the cockpit terminal is in a monitoring state; a state synchronization control unit, configured to reversely control the simulation control device according to a control instruction of the simulation control device so as to synchronize the state of the simulation control device with the state of the vehicle; The device further comprises: a first auxiliary control unit, configured to, when determining the current state of the cockpit terminal, control the simulation control device to maintain the current state until receiving new vehicle state data if the cockpit terminal switches from the remote control state to the monitoring state; The monitoring state refers to a state in which the remote driver does not control the autonomous vehicle, and the remote control state refers to a state in which the remote driver issues a vehicle control command to the autonomous vehicle at the cockpit end, causing the autonomous vehicle to enter a parallel driving mode from an autonomous driving mode.

8. A parallel driving system, characterized in that: It includes a vehicle side, a service side and a cockpit side; the vehicle side is used to upload the vehicle data to the service side, and the vehicle data includes the vehicle status data; The server is used to send the vehicle data to the cockpit; The cockpit end is used to execute the parallel driving assistance method as described in any one of claims 1-6.

9. The parallel driving system according to claim 8, characterized in that: The vehicle-side is configured to generate a remote takeover request based on a fault detection result of a vehicle-side fault self-diagnosis program; and / or obtain a decision instruction generated by a vehicle-side autonomous driving program and generate a remote takeover request based on the reliability of the decision instruction; and / or generate a remote takeover request when the vehicle-side is in a distressed state due to an unexpected stop during autonomous driving; Uploading the remote takeover request to the server; The server is configured to send the remote takeover request to the cockpit; The cockpit end is used to receive the remote takeover request, generate and display a remote control task according to the remote takeover request.

10. The parallel driving system according to claim 9, wherein: The cockpit side is used to generate remote vehicle control instructions based on the displayed remote control task, the three-dimensional motion model centered on the vehicle, and the vehicle-side monitoring video, and send the remote vehicle control instructions to the server side; The server is used to send the remote vehicle control command to the vehicle; The vehicle end is used to receive the remote vehicle control command and switch to the remote driving mode.

11. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the parallel driving assistance method according to any one of claims 1 to 6.

12. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, enable the electronic device to execute the parallel driving assistance method as described in any one of claims 1 to 6.

Images